/****************************************************************************************/
//
// NAME: README
// AUTHOR: Nicolas HAHN
// COPYRIGHT: Copyright (c) 1999-2015 Nicolas HAHN <hahnn@x-itools.com>
// DATE OF CREATION: 11/04/2013
// PROJECT: X-Itools
// MODULE: X-Itools mail management web client
// DESCRIPTION: this is the README file. It gives a small status about
// versionning of the E-mail Log Search Engine tool
// LICENCE: GNU - GPLv3. to have more information about the licence,
// read the LICENCE file
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
//
/****************************************************************************************/
IMPORTANT!!! READ THE OFFICIAL DOCUMENTATION!!!
-----------------------------------------------
The official documentation of the X-Itools Mail Manager Tool (called ELSE for Email
Log Search Engine) is available on the SourceForge project page, in the Wiki.
https://sourceforge.net/p/x-itools/wiki/
PLEASE READ IT!!!
PURPOSE OF THE X-ITOOLS MAIL MANAGER TOOL
-----------------------------------------
This tool is designed for strong Postfix E-mail tracking, filtering, threats detection
and counter-measures. It is also able to parse a lot of different Microsoft Exchange
Server Log files.
It is composed of a server part, and a client part.
The server part is simply the Postfix + Rsyslog + PostgreSQL database combination.
The client part is a rich web 2.0 application made using the ExtJS 4.1.3 framework.
Client part is currently not working with another version of ExtJS than the 4.1.3.
Basically, Postfix configuration needs some small adaptations for this tool,
as well as Rsyslog configuration adaptations.
The way it works, basically:
- Postfix or the RSYSLOG WINDOWS AGENT in the case of Exchange Server 2010 log parsing
send all logs to the central RSyslog server
- the central Rsyslog send all logs directly to the PostgreSQL database in a specific
table
- A trigger is defined on this table, and as soon as a tuple is inserted, a set of
PgSQL stored functions are called to read, decode, process and re-arrange the
Postfix log line in all the other tables.
- Then anyone can use the Web application to query the database and track any
e-mail flow.
The 2004 version of this tool was not real time. It used a parser written in C, asking
the maillog file (in /var/log) to process.
Today's version of the tool is real time. Some milliseconds after an e-mail has passed
through your postfix servers, you will be able to query for it in the Mail Manager
tool. Of course, the "real time" ability is directly linked to the processing power of
your database server.
See the INSTALL file for more information.
If you plan to use this tool for Microsoft Exchange Server 2010 log file parsing, you
MUST also read the EXCHANGE_SRV_LOGS readme file.
IMPORTANT: for now this tool is just R&D. I don't provide so called PRODUCTION releases
or STABLE ones. I provide what I like to call INNOVATION releases only until I reach a
certain level of features I've in mind and to be implemented. I also use this project
to explore PostgreSQL DB capabilities and features, and to learn PHP and ExtJS by
myself. That means I don't have best coding practices in mind at this time and until
further notice: this is clearly not the best PHP, Javascript and SQL code you'll
never see in your life. So in clear, USE THIS OPEN SOURCE SOFTWARE AT YOUR OWN RISKS!
Also in a not so far futur, I might choose another OPEN SOURCE LICENSE to have the
software still Open Source, but to have paying customers for the product. I'm thinking
about it. Currently this software is GPLV3. If you need professionnal support,
consulting, etc. about this software, feel free to get in contact with hahnn@erios.org
or hahnn@x-itools.com.
HISTORY
=======
0.9.19:
-------
Summary:
- IP Addresses geographic location implemented. GeoLite2 databases, provided by MaxMind, are now integrated into the ELSE database. When you perform a search, details about sending server location are displayed in the search result window. This is also all integrated with Goole Maps.
- Addition of the "IP Locator" feature in the "Test tools" tab of the ELSE WUI: give any IP address, and it will show you where it is located using Google MAP.
- The ELSE now handles SPF policies and rules. That can be used to influence SPF decisions provided by the GreyLSE SPF engine. SPF Policies and rules are managed in the ELSE Web user Interface, and SPF decision results can also be used in other policies and rules, like the ones from the Blacklist or Whitelist for example. Also, a lot of different SPF reports are provided under the "Statistics" tab of the ELSE WUI.
- The GreyLSE Postfix Policy Server has now a built-in SPF engine: it is now able to make SPF checks
- Postfix/dnsblog are now parsed by the ELSE. By the way, several DNSBL reports are now available under the "Statistics" tab of the ELSE WUI.
- The GreyLSE Postfix Policy Server has been updated to be able to understand Postfix V 3.x Policy Delegation Protocol
Details:
(Trunk r2421):
- updated README file for ELSE v 0.9.19
(Trunk r2420):
- updated GreyLSE INSTALL file: added a content table and details for Systemd based Linux distributions
(Trunk r2419):
- added var/run/greylse path and updated makefile and configure
(Trunk r2418):
- updated makefile and configure for the GreyLSE
- updated GreyLSE for SQL schema version 960
(Trunk r2417):
- update of some makefile.am files
(Trunk r2416):
- updated GreyLSE.C to drop its privileges after having written the pid file
- GreyLSE daemon now write its pidfile in /var/run/greylse/greylse.pid
- updated greylse SysV initscript with new pidfile path
- added systemctl greylse configuration file
(Trunk r2415):
- typo fixes in the INSTALL file
(Trunk r2414):
- added a content table to the INSTALL file, added GOOGLE MAPS API and GEOIP sections, updated the CREATION OF THE DATABASE section to include PostgreSQL timezone configuration to UTC (the database cannot run if your PostgreSQL cluster is not configured with 'UTC' timezone)
(Trunk r2413):
- Added the "IP Locator" tool in the ELSE WUI, under the "Test tools" tab: give an IP Address, and it will display its location on a Google MAP
(Trunk r2412):
- updated index.html to include Google Maps APIs
- Sending Server IP address location is now displayed on a Google Map included in the Search Results Window of the ELSe WUI, in the "Connecting Client" field set
(Trunk r2411):
- added LIMIT 1 to the SQL request executed in getlocation() SQL method because we know this SQL request must return only one line and it's the way to obtain best performances (5 times faster). Please note that this perf boost is obtained when using PostgreSQL 9.4 and if a GIST index is available on the network column of the geoip.blocks SQL table
- updated XItool_MailManager_0.9.18_0.9.19.out to add the command needed to build a GIST index on the network column of the geoip.blocks SQL table. However, we commented out the CREATE INDEX instruction because it can be used only if your DB backend is PostgreSQL 9.4. If it is the case, uncomment it and after the GIST index is created, drop the standard B-tree index that is existing on the column, which is useless
- an idea of performance boost otained from PostgreSQL 9.3 (with standard index on network column of geoip.blocks table) to PostgreSQL 9.4 (with a GIST index instead):
9.3: getlocation() SQL method exec time: 789.648 ms
9.4: getlocation() SQL method exec time: 16.206 ms
(Trunk r2410):
- added getlocation() SQL method. This method takes a value of type inet in parameter, query the GeoIP SQL tables in the geoip schema, and returns the answer as a table (1 line returned)
- provided XItool_MailManager_GeoIP_0.9.19.out.gz file: this file is compressed using GZIP. It contains the GeoIP tables, that will be in geoip schema. When uncompressed, the size is 261 Mb. This uncompressed file has to be installed ONCE in the ELSE (starting from version 0.9.19) PostgreSQL database using the way ELSE database patch files are deployed. Once installed, the getlocation() SQL method can be used to provide geographic IP location.
- this commit is a needed step in order to get geographic IP location implemented in the ELSE WUI: soon, you'll know not only the IP of client SMTP servers connecting to your Postfix servers, but also where they are located.
(Trunk r2409):
- added some reports about DNSBL, available under the "statistics" tab of the ELSE WUI
(Trunk r2408):
- added dnsbl_codes, dnsbl_providers and dnsblog SQL tables
- implemented SQL methods to parse postfix/dnsblog log lines
- updated trigger_parser() and cleanner() SQL methods to enable postfix/dnsblog parsing and cleaning of dnsblog table
(Trunk r2407):
- updated INSTALL file with the latest list of PHP RPM packages
- updated dnsresolver.php to make use of system configured DNS servers (from /etc/resolv.conf) instead of hard coded '127.0.0.1' name server. Please note that php-pear-Net-DNS2 must be installed on the ELSE server to get DNS name resolution working.
(Trunk r2406):
- implemented several tens of SPF reports, available under the "statistics" tab of the ELSE WUI
(Trunk r2405):
- bug fix: in greylse_spf_enabled() SQL method, a bad condition in the last select SQL request (wrong column tested: greylse_id instead of client_id) prevented the SPF engine to work correctly
- added spf_logs SQL table: it will record all decisions made by the SPF engine, allowing the generation of statistics/reports that will be implemented soon
- updated greylse() SQL method to make use of new spf_logs SQL table
(Trunk r2404):
- in fact, errors table was already cleaned...
(Trunk r2403):
- created an index on logs(date)
(Trunk r2402):
- updated cleanner() SQL method to clean errors, rtaam_console and logs SQL tables
(Trunk r2401):
- bug fix: in trig_smtp() SQL method, when parsing rcpt_to field, limit length to 255 characters, to avoid blocking DB if a user send an email to malformed recipient email addresses (had the case yesterday with a user sending to "adr1@dom1.com,adr2@dom2.com,adr3@dom3.com,..." because he used "," instead of ";" separator in thunderbird...)
- bug fix: the "order" columns in the whitelist and blacklist grids of the ELSE WUI should now be correct. The issue was due to a mistake in some SQL requests used when adding the blacklist or whitelist rules. Please note that if you've existing rule before this fix, their order will not be updated.
(Trunk r2400):
- Bug fix: SPF_PASS policy was not enforced in the greylse() SQL method due to a wrong condition evaluation
(Trunk r2399):
- it's now possible to start/stop a GreyLSE instance using the ELSE WUI. SNMP, your firewalls, ... have to be configured accordingly for this feature to work:
- updated startcmd and stopcmd default values, in greylses SQL table
- updated "GreyLSEs" sub-tab of "Manage" tab: stop and start buttons now call scripts to effectively start or stop the selected GreyLSE instance
- provided startgreylse.php and stopgreylse.php scripts, that make a call to snmp_exec script
(Trunk r2398):
- Bug fix: fixed some wrong directory paths as well as calls to snmp* scripts outside of php directory
(Trunk r2397):
- implemented handling of SPF rules in greylse() SQL method. SPF check result from the SPF engine can now be influenced by the SPF rules that can be defined in the ELSE
(Trunk r2396):
- replaced 'spf' by 'spfcr' to avoid replacement mistakes in the user defined rules
(Trunk r2395):
- added the "SPF Rules" sub-tab under the "WGB Lists" tab of the ELSE WUI. It's now possible to manage SPF rules in the ELSE. Those rules will be enforced by the GreyLSE engine.
- created spf_rules SQL table to record SPF rules
(Trunk r2394):
- updated greylse() SQL method, GreyLSE daemon and ELSE WUI to handle new commands provided in Postfix Policy Delegation Protocol version 3.0
(Trunk r2393):
- updated the various dialog boxes used to manage GreyLSE rules (white/black/grey/hold listing rules) to allow the use of the new SPF variable: this variable can be used to take car of the SPF check results in GreyLSE policy rules
- updated greylse() SQL method to take care of the SPF variable in GreyLSE rules, and also to start to implement dedicated SPF policy management, with dedicated SPF rules
(Trunk r2392):
- Implemented SPF engine in the GreyLSE daemon.
- SPF checks are performed via a dedicated GreyLSE child.
(Trunk r2391):
- Bug fix: a mistyped symbol in the TIMEOUT variable name prevented the "stop" command of the GreyLSE startup script to work properly
(Trunk r2390):
- updated compile script of greylse daemon to include the libspf2 library
- added column disable_spf to greylse_clients SQL table
- updated wui_greylseclientslist() SQL method to provide disable_spf column in the result set
- updated greylseclientslist.php and updategreylseclient.php to take of SPF flag
- updated README and INSTALL files for greyle, because libspf2 will be used
- updated the ELSE WUI, in "Manage" tab, "GreyLSEs" sub-tab, when selecting a GreyLSE server and displaying client Postfix servers, the clients windows shows the new SPF flag
(Trunk r2389):
- updated all necessary files for the PHP scripts path
(Trunk r2388):
- moved include directory into php directory
(Trunk r2387):
- moved etc directory into php directory
(Trunk r2386):
- moved all php scripts into php directory
(Trunk r2385):
- created php directory
(Trunk r2384):
- updated comments at head of file cleanlist.php
(Trunk r2383):
- moved jquery libraries in include/ folder, updated index.html and INSTALL file for new path
- added some missing monitoring scripts in Tools/Monitoring folder, in particular CheckMaps.sh that is needed by the ELSE
0.9.18:
-------
Summary:
- RTAAM is now available. It means Real Time Alerting and Action Module. This is a very important and big feature implemented in ELSE v0.9.18. To give you a picture, we could try to define it like this: RTAAM is to be seen as an E-mail flow threats detection and prevention system. Another definition could be: RTAAM is an E-mail Firewall solution. Or let's give a last definition: RTAAM is a mix of "Postfix/Anvil", "Fail2ban", "Firewall", "Monitoring", "Reporting". The RTAAM engine is able to work in near real time (NRT), because it's able to dynamically take its decisions in a 1 minute time frame minimum. With RTAAM, you can do things like identify internal user workstations infected by SPAM bots and of course you can block any email traffic generated by them. You can define policies, containing rules written using a lot of different parameters, including the ones made available by the ELSEre (ELSE reputation engine). You can implement a set of quotas on your email flows. Or you can just get and draw statistics about any individual e-mail address or IP address connecting to your Postfix servers. And more. RTAAM is an important complement to an already existing "security" engine which is the GreyLSE. And, RTAAM active features (understand: protection features) are managed by the GreyLSE itself.
- Implemented compatibility of the ELSE VirtualHost with Apache 2.4 version using mod_authn_dbd
- The ELSE user, if he is administrator, is requested to select if he wants to use the ELSE or the ELSEMC at application loading time
- Started code refactorisation: moving PHP code in SQL methods as much as possible to avoid keeping logic in PHP scripts and increasing PHP code robustness by making use of PDO prepared statements
- GreyLSE now log SQL request in syslog if it failed
- In GreyLSE, implemented auto-whitelisting of recipient e-mail addresses (RAWL) when your users send them emails, in order to avoid your users contacts to be catched by GreyLSE filtering when they send answers. This feature can be enabled by your users using the ELSEMC
- postfix/postscreen logs are now parsed by the ELSE, and it's possible to get statistics about Postscreen in the ELSE WUI
- Implemented ELSEre (ELSE Reputation Engine) reporting: top 10 to top 50 e-mail addresses reputations, for senders and recipients. They appear like "age pyramids". The user can generate those reports by selecting the domains he wants.
- We've started to prepare the things to process APACHE web server logs in the near futur
- Drastically reduced the number of blocking SQL calls in the ELSE WUI. That means several SQL requests can be launched in parallel in the ELSE WUI: for example, performing a search taking a lot of time will not block the user in making other actions from the point of view of the database.
- Attachments without any specified MIME types are now handled and displayed in the email details window of the ELSE WUI
- Optimizations at database level when processing QMGR log lines: new qmgr_attempts SQL table created, old qmgr SQL table not used any more and will be removed later. But WARNING: previous content of qmgr table is not transfered in qmgr_attempts table, that mean you'll not get access to data stored in the old qmgr table via the ELSE!
- Started to implement best coding practices by using OWASP PHP ESAPI libraries
- Bug fix: the number of attachments of found emails in the search results window is now correctly displayed.
- Bug fix: fixed an issue preventing the list of email sender addresses to be displayed in the combobox of the "E-mail queue" window.
- Bug fix: updated the backward search interval from 5 days to 5 days and 1 hour in the trig_qmgr() SQL method to avoid some potential database crashes. Please note this interval is set considering your postfix servers will not keep trying to send emails after 5 days.
- Bug fix: entering in an endless loop in the trig_qmgr() SQL method was possible, in rare conditions: if a qmgr log line has a status defined. That should be fixed now.
- Bug fix: in the "Manage" tab, "Billing reports" sub-tab; When running a report, sometimes, the graphs were not displayed. It was due to anissue in an SQL request in wui_loadReportData() SQL method
- Bug fix: so called "billing reports" graphs were wrongly rendered in the case the e-mail traffic was really low, because the graph was generated only when there were data.
- Bug fix: in the window showing the details of an e-mail, the "queued as" column was empty due to an issue in the SQL statement.
- Bug fix (try): the database could enter in endless loops when processing the trig_qmgr() SQL method, only in this case: when you have a postfix server with emails in queue (defer queue) and if you stop then start again this postfix server. In this case, the PID of the qmgr process (as well as all others) changes and is consequently not the same than the one of previous e-mail transactions for a same email. In this ELSE release, several fix_qmgr() methods are provided to try to deal and fix emails in such situation, and the trig_qmgr() SQL method has been updated to try to avoid such situation to occur again. But I'm not sure that will be enough, that's why this bug fix is just a try for now.
- Bug fix: in the "Billing report" or "RTAAM report" graphs, the vertical axis had several times the same numeric value in case data to be displayed had very small values. Now, the vertical axis displays floats to avoid such behavior.
Details:
(Trunk r2382):
- updated README file for ELSE v 0.9.18
(Trunk r2381):
- updated for greylse to use DB shema version 959
(Trunk r2380):
- forgot to put HoldlistRulesStore.js on SVN repository
(Trunk r2368 to r2379): REORGANIZATION OF SVN REPOSITORY STRUCTURE
(Trunk r2367):
- updated README file for ELSE v 0.9.18
(Trunk r2366):
- renamed the original X-itools directory (20010715_01) to EnterpriseCollaboration. This contains the original project which is not developed any more.
(Trunk r2365):
- moved the database script files of the ELSE sub-project in their new home directory
(Trunk r2364):
- Moved the directory of the ELSE (E-mail Log Search Engine) sub-project at the root directory of the repository
(Trunk r2363):
- in trig_qmgr() SQL method, encapsulated inserts into qmgr_attempts SQL table in an exception catch block to capture unique violations
(Trunk r2362):
- Bug fix: in the "Billing report" or "RTAAM report" graphs, the vertical axis had several times the same numeric value in case data to be displayed had very small values. Now, the vertical axis displays floats to avoid such behavior.
(Trunk r2361):
- INSTALL file updated for ELSE version 0.9.18
(Trunk r2360):
- generates the Excel file to export RTAAM statistics
(Trunk r2359):
- in the RTAAM reporting window, set time of the timefield to '00:00' by default; it was set to the nearest current hour previously, but it seems a bug in the EXTJS framework v 4.1.3 prevent this timefield to work correctly when the RTAAM reporting window is called and displayed more than once
- config.php file switched to ELSE version 0.9.18
(Trunk r2358):
- bug fix: when doing extended searches in the ELSE, the number of attachments was almost never exact in the window displaying results, remaining to 0: that was due to algorithmic issues in the removeattachmentswaitarea() SQL method that have been fixed now.
(Trunk r2357):
- implemented RTAAM reporting GUI in the ELSE: select a policy, then double-click on it. The RTAAM reporting window will be displayed
- The RTAAM reporting window will show all statistics collected on matching rules, for individual IPs or Sender e-mail addresses
- RTAAM statistics data grid can also be displayed
(Trunk r2356):
- added wui_loadRtaamReportData() SQL method, to provide statistics about RTAAM activities. Those stats can then be graphed in the RTAAM reporting GUI
(Trunk r2355):
- implemented RTAAM temporary blocking feature in the greylse() SQL method, which means RTAAM reject rules are now really enforced. Sending IP addresses and RFC821 Sender email addresses can be temporarily blocked now. If a REJECT rule match, a defer_if_permit 4.7.1 SMTP code is sent.
(Trunk r2354):
- Added a "Monitoring" button in the RTAAM console. clicking on it will display the RTAAM Real Time monitoring window.
- This window displays status of any IP address or RFC821/822 sender e-mail address on which the RTAAM engine made an action (WARNING, REJECT or BLOCK). All information are splitted between 9 grids that are refreshed every minute. So at any time, you know if E-mail addresses or IP addresses are blocked according what you've set in the RTAAM rules, in fact you know if there is an attack on your e-mail flows.
(Trunk r2353):
- added wui_rtaammonlist() SQL method: this method provides the list of IP addresses or Sender e-mail addresses in a warning, reject or block state from RTAAM engine
(Trunk r2352):
- implemented RTAAM permanent blocking feature in the greylse() SQL method, which means RTAAM blocking rules are now really enforced and RTAAM engine become an active security feature. Sending IP addresses and RFC821 Sender email addresses can be permanently blocked now.
- Revised trig_qmgr() SQL method again and implemented 3 different fix_qmgr() SQL methods to try to provide a fix to the situation below:
- let's say you have emails in the defer queue of your postfix servers since 2 days, due to a postfix configuration mis configuration for instance
- then you fix the configuration but have to restart (not reload) Postfix
- then Postsfix will proceed to next delivery attemps of emails stored in defer queue
- of course logs are catched by the ELSE, by the database
- but suddenly you don't see new emails available in the ELSE WUI
- you check the databse logs, and you see it cannot process Postfix logs any more, your database log files are growing fast because of SQL errors
- what's the problem there?
- in fact, the qmgr_uniqinfos table and the trig_qmgr() SQL method used to parse Postfix QMGR log lines take care of the Postfix qmgr daemon PID
- when you've restarted Postfix, of course, for the same emails processed by the various Postfix daemons, the PID of them have changed
- this lead to the insertion of additional lines in the qmgr table for the same email QIDs, generating SQL errors in the trig_qmgr() SQL method
- what's a possible solution?
- in the qmgr SQL table (and some others), the various PIDs must be reconciliated in a way or another in order to get a unique line inserted in the qmgr table for a given QID
- we cannot enforce this by a UNIQUE condition or a primary key on the qmgr table, because we now that the Postfix QIDs are not unique especially if you have plenty of different Postfix servers, so yes, it's possible to get several times the same QID inserted in the qmgr table
- to be able to enforce this condition, a more complex algorithm must check it, taking care of other email elements like dates of attempts, number of recipients and so on, those elements being used to be sure to retreive the exact qmgr table record we need
- then for such issues (the restart of a Postfix server that had emails in queue), it has been decided to simply keep the first original PID recorded in the SQL table, as if Postfix server would never have been restarted
- IN CASE YOUR DATABASE IS IN THIS SITUATION, WHERE IT'S BLOCKED, you can manually use the fix_qmgr() SQL methods, and we recommend to use the one taking no parameters; By using a SQL request in psql like "select fix_qmgr();" what will happen is that all "duplicates" of the qmgr SQL table will be retreived over the past 6 days, and this method will try to fix them.
- this will unblock the parsing of Postfix log lines by the database and should stop to make your database log files growing fast. If it's not the case, it means your issue is not the one described here :)
- in the mean time, we've tried to update the trig_qmgr() parsing method in order to take this particular situation in consideration, in order to not make the use of fix_qmgr() methods a need. Maybe the updated done in trig_qmgr() will still not be enough and we're going to make further testing about that.
(Trunk r2351):
- bug fix: in the ELSE WUI, in the details of an e-mail transaction, in the list of recipients showing status when email is sent to next server, the "queued as" column of the grid was empty. This was due to an issue in the SQL statement, which is now fixed. "Queued as" information column now displays the right information again.
(Trunk r2350):
- added the "RTAAM BL" sub-tab under the "RTAAM" tab in the ELSE WUI
- This tab show a list of IPs and RFC821 E-mails automatically blacklisted by the RTAAM engine
- Entries in this list will be managed by the GreyLSE to implement the ACTIVE mode of the RTAAM
(Trunk r2349):
- added new rtaam_bl SQL table: IP addresses or RFC821 email addresses to be blocked are recorded in it by the RTAAM engine. And this table will be used by the GreyLSE to apply the blocking
- updated rtaam() SQL method to record items in rtaam_bl SQL table, if rule of policy is to BLOCK
(Trunk r2348):
- a status about the RTAAM engine: it's now operationnal as a PASSIVE feature at this time. That means it's able to detect behaviors on e-mail flows according to your policies/rules, and detected events are sent to the RTAAM console as logs: this is equivalent to saying RTAAM engine is able to run in REPORTING ONLY mode for now)
- in the ELSE WUI, the window for graphing billing statistics has been enhanced a little bit: when the "Auto-refresh" button is clicked, data for the graph are immediately reloaded without waiting the 20 first seconds, and they are reloaded according settings made in the window: it avoids the need to click on the Refresh button to take new settings in consideration before to click on the "Auto-refresh" button
(Trunk r2347):
- added rtaam_senders_emails_stats and rtaam_senders_emails_status SQL tables: they will keep track of e-mail addresses in a warning, reject or block state
- implemented logic based on RFC821 and RFC822 sender email addresses in RTAAM engine
- reviewed wui_loadReportData() SQL methods to really provide data every SAMPLES minutes, and not only every SAMPLES minutes when there are records (making holes in the graphs of the WUI)
(Trunk r2346):
- it should now be possible to export RTAAM engine logs to EXCEL format
(Trunk r2345):
- enabled the "Console" sub-tab of the "RTAAM" tab: this console will show all warning, reject and block alerts of the RTAAM engine. It's a good idea to not forget to enble it's auto-refresh button. Have to implement the download button next time.
(Trunk r2344):
- continue coding rtaam() SQL method
- added rtaam_senders_ips_status SQL table: this table will keep track of IP addresses in a warning, reject or block state
- added wui_rtaamconsolelist() SQL method: this provides logs of the RTAAM to be displayed in the "Console" tab
(Trunk r2343):
- added a new entry in postgreSQL crontab to execute RTAAM engine every minute
(Trunk r2342):
- continue to implement all what is needed for RTAAM engine, from the point of view of the ELSE WUI
- it should now be possible to manage RTAAM policies and there rules using the ELSE WUI, in "RTAAM" tab, "Policies" tab.
(Trunk r2341):
- bug fix: in a wui_loadReportData() SQL method, a wrong SQL variable name was used, and caused an SQL error, preventing billing graphs to be displayed for a specific domain in the GreyLSE WUI
- added rtaam_console SQL table: this table will record RTAAM events for history
- added rtaam_senders_ips_stats SQL table: this table will record statistics about rules that match in RTAAM policies
- added rtaam_rules SQL table: this table will record RTAAM rule definitions for RTAAM policies
- added rtaam() SQL method: this is the RTAAM engine. Of course, coding of this method is just starting and is not fully implemented
- added wui_rtaamrulelist() SQL method: this method will provide the list of rules in a RTAAM policy to be displayed in the WUI
(Trunk r2340):
- added new target column to rtaam_policies SQL table
- added new senderiptraffic_monitor SQL table: this table will provide statistics about IP addresses of sending hosts, and that will be used by RTAAM
- updated bill_r_stats() SQL methods to generate statistics in senderiptraffic_monitor SQL table
- added new rtaam_senders_ips() SQL method: it generates RTAAM statistics about sending IP addresses returned as a SQL table when called
(Trunk r2339):
- in RTAAM tab, added column "Target" to list of policies. A RTAAM policy can be applied to RFC821 Sender email addresses, RFC822 ones, or to sending IP addresses.
- Updated other WUI dialogs (Addition of a RTAAM policy, edition), as well as the store
- Updated rtaampolicylist.php, updatertaampolicy.php, addrtaampolicy.php accordingly
(Trunk r2338):
- created new rtaam_policies SQL table
(Trunk r2337):
- started to implement RTAAM features: Real Time Alerting and Action Module
- enabled RTAAM tab in the ELSE WUI
- implemented models, stores, WUI components and PHP scripts to allow adding, updating and deleting RTAAM policies
- next step is to focus on the RTAAM rules ditor
(Trunk r2336):
- bug fixed: in trig_qmgr() SQL method, entering in an endless loop was possible, in rare conditions: if a QMGR log line has a status defined. In such case, the method wasn't able to retreive the original QMGR log lines in the qmgr_uniqinfos table because of a lack of information (size=, nrcpt= fields not there)
(Trunk r2335):
- in trig_qmgr() SQL method, updated an interval of 5 days to an interval of 5 days and 1 hour in a backward search operation. In fact, if postfix keeps trying to send the email during 5 days, we can have logs for a little bit more than 5 days to process. And without this interval update, the database may crash
(Trunk r2334):
- updated dosearch.php and drawmailflow.php to make use of new qmgr_attempts SQL table instead of qmgr SQL table
(Trunk r2333):
- added new qmgr_attempts SQL table. This table comes in replacement of the qmgr SQl table: the benefit is this new table is smaller because all duplicated information (also in qmgr_uniqinfos SQL table) have been removed. It should be faster also. Old qmgr table (as well as children tables) will be removed later
- updated trig_qmgr(), trig_postsuper(), rebuildlogs() and partition_cleaner() SQL methods to make use of new qmgr_attempts SQL table
(Trunk r2332):
- updated trig_cleanup() SQL method to handle attachments without specified MIME types. In such case, an 'unknown/unknown' MIME type is set.
(Trunk r2331):
- added new wui_fullsearch() SQL method
(Trunk r2330):
- dofullsearch.php and exportfullsearch.php rewritten to call wui_fullsearch() SQL method and make use of PDO prepared statement
(Trunk r2329):
- added new wui_loadautocleanreport() SQL method
(Trunk r2328):
- exportautocleanreport.php and loadautocleanreport.php rewritten to call wui_loadautocleanreport() SQL method and make use of PDO prepared statement
(Trunk r2327):
- added new wui_elserereportslist() SQL methods
(Trunk r2326):
- elserereportslist.php rewritten to call wui_elserereportslist() SQL method and make use of PDO prepared statement
(Trunk r2325):
- added new wui_attacheddomains4elserelist() SQL method
(Trunk r2324):
- attacheddomains4elserelist.php rewritten to call wui_attacheddomains4elserelist() SQL method and make use of PDO prepared statement
(Trunk r2323):
- added new wui_loadReportData() SQL methods
(Trunk r2322):
- exportreportdata.php rewritten to call wui_loadReportData() SQL method and make use of PDO prepared statement
- loadreportdata.php rewritten to call wui_loadReportData() SQL method and make use of PDO prepared statement
(Trunk r2321):
- exportelserereportdata.php rewritten to call wui_loadELSEreReportData() SQL method and make use of PDO prepared statement
(Trunk r2320):
- loadelserereportdata.php rewritten to call wui_loadELSEreReportData() SQL method and make use of PDO prepared statement
- elserechartdomainslist.php rewritten to call wui_elserechartdomainslist() SQL method and make use of PDO prepared statement
(Trunk r2319):
- added new wui_elserechartdomainslist() SQL method
- added new wui_loadELSEreReportData() SQL method
(Trunk r2318):
- sendersqueuelist.php updated to make use of OWASP libraries
(Trunk r2317):
- exportqueuelist.php rewritten to call wui_queuelist() SQL method and make use of PDO prepared statement
- queuelist.php and exportqueuelist.php updated to make use of OWASP libraries
- added SMTPSrvName.php OWASP validator class to validate inputs requested by exportqueuelist.php and queuelist.php
(Trunk r2316):
- added a section about OWASP. version 0.9.18 of the ELSE will rely on OWASP PHP ESAPI libraries for good coding practices and security of web applications
(Trunk r2315):
- added new wui_checkperms() SQL method.
(Trunk r2314):
- checkperms.php rewritten to call wui_checkperms() SQL method and make use of PDO prepared statement
(Trunk r2313):
- added new wui_sendersqueuelist() SQL method.
(Trunk r2312):
- sendersqueuelist.php rewritten to call wui_sendersqueuelist() SQL method and make use of PDO prepared statement
- fixed column naming according result sets returned by DB in queuelist.php
(Trunk r2311):
- added new wui_queuelist() SQL method.
(Trunk r2310):
- queuelist.php rewritten to call wui_queuelist() SQL method and make use of PDO prepared statement
(Trunk r2309):
- bug fix: the list of email sender addresses were not displayed any more in the combobox of the "E-mail queue" window. That was due to a change in the structure of the pfx_cache SQL table in ELSE v0.9.17, where server column passed from character varying to smallint type, and because the sendersqueuelist() SQL method was not updated accordingly. This SQL method is now updated.
(Trunk r2308):
- serverlist.php rewritten to call wui_serverslist() SQL method and make use of PDO prepared statement
(Trunk r2307):
- added new wui_serverslist() SQL method
(Trunk r2306):
- make use of session_write_close() to enhance user experience in the WUI. Actions can be done in parallel: for example, a full search operation can be started and may need several minutes to provide results, and at the same time another full search can be performed with different search criterias and may provide results before the end of the first full search action.
(Trunk r2305):
- added new wui_userinfo() SQL method
(Trunk r2304):
- userinfo.php rewritten to call wui_userinfo() SQL method and make use of PDO prepared statement
(Trunk r2303):
- make use of session_write_close() to enhance user experience in the WUI. Actions can be done in parallel: for example, a full search operation can be started and may need several minutes to provide results, and at the same time another full search can be performed with different search criterias and may provide results before the end of the first full search action.
(Trunk r2302):
- added new wui_qidslist() SQL method
(Trunk r2301):
- qidslist.php rewritten to call wui_qidslist() SQL method and make use of PDO prepared statement
- fixed the root flag to false when calling wui_attachmentslist() SQL method (otherwise an ELSE administrator only can get the results)
(Trunk r2300):
- added new wui_attachmentslist() SQL method
(Trunk r2299):
- rewritten to call wui_attachmentslist() SQL method and make use of PDO prepared statement
(Trunk r2298):
- updated header of source files
- added ExtJS required components at beginning of MyBillsGridPanel.js
- bug fix: in file attachmentslist.php, there was a mistake at the end of the file (some forgotten lines of an old version for this file), after the "?>" line, preventing the attachments to be displayed in the Search Results window
(Trunk r2297):
- added new 'ELSEre' sub-tab under the 'Manage' tab of the ELSE Web user Interface
- in the 'ELSEre' sub-tab, it's possible to create new reports to generate statistics about data provided by the ELSE Reputation Engine
- Reputation statistic are provided by the way of two charts: one for senders and the other for recipients. Those graphs are like "age pyramid". For every graph is shown the worst and best reputations.
- It's possible to display the data as a grid, and to export this grid as an Excel file
(Trunk r2296):
- added generateELSEreReport() SQL method: this method is in charge of the creation of the data for ELSE Reputation Engine statistics
(Trunk r2295):
- Added elsere_domaingroups and elserereports SQL tables. Those tables are used to implement the ELSE Reputation Engine reports and statistics
(Trunk r2294):
- added header of comments to several *.js files
(Trunk r2293):
- renamed APACHE_LOGS file as APACHE_SRV_LOGS
(Trunk r2292):
- added new httpd_systemevents SQL table: it will receive Apache access log lines
- added trigger empty function trigger_httpd_parser() for this table
(Trunk r2291):
- updated to include a dedicated rsyslog PostgreSQL template for Apache HTTPD logs processing
(Trunk r2290):
- typo fixes
(Trunk r2289):
- APACHE_LOGS text file contains documentation usefull to configure your apache httpd web servers as well as other components of the ELSE server (like Rsyslog) to allow the ELSE to parse Apache Access log files information.
(Trunk r2288):
- updated trig_postscreen() SQL method to process postfix/postscreen NOQUEUE rejects due to postscreen errors like "too many connections" and "all server ports busy"
- those errors, if they occur, will be displayed in the "Postfix warning/fatal/panic" window of the ELSE wui as fatal messages
(Trunk r2287):
- added cleaning of postscreen SQL table partitions in cleaner() SQL method
(Trunk r2286):
- added an additionnal report about postscreen
(Trunk r2285):
- added 14 new reports dedicated to postscreen, available in the "statistics" tab of the ELSE web user interface.
(Trunk r2284):
- added column infos to postscreen table; That will record parsed text that are at the end of some postscreen parsed log lines: updated trig_postscreen() SQL method for that.
- added a first, basic, postscreen report in the reporttypes SQL table (with id 35). This report, called "Postscreen statistics", is available in the "statistics" tab of the ELSE Web user interface.
- added queryroot and queryuser columns in table reporttypes: those column now contain the SQL request to be executed by the getstats() SQL method used to generate and display statistics in the "statistics" tab of the ELSE web user interface.
- rewritten getstats() SQL method, that now read (from reporttypes SQL table) and execute the SQL request for the selected report
(Trunk r2283):
- added new postscreen_partition() SQL method to enable table partitioning as well as a trigger for postscreen table
(Trunk r2282):
- added new command column to postscreen SQL table
- updated trig_postscreen() SQL method to parse the "command" part of the log line when relevant
(Trunk r2281):
- created postscreen SQL table, to record all postscreen activity
- updated trig_postscreen() SQL method: we now have parsing of postscreen logs operationnal in its first version. All data stored in the postscreen SQL table. What remains to do is reports and statistics in the web interface.
(Trunk r2280):
- created SQL table postscreen_types. It will record all postscreen operation types like: WHITELISTED|BLACKLISTED|PASS OLD|PASS NEW|WHITELIST VETO|PREGREET|DNSBL|COMMAND PIPELINING|NON-SMTP COMMAND|BARE NEWLINE|HANGUP|COMMAND TIME LIMIT|COMMAND COUNT LIMIT|COMMAND LENGTH LIMIT
- updated trig_postscreen() SQL method to get the type of the postscreen operation
(Trunk r2279):
- modified a test about hostname and ip address of connecting client in trig_postscreen() SQL method, to avoid an issue with a NULL column constraint if hostname is not defined
(Trunk r2278):
- added trig_postscreen() SQL method. This method will contain all the logic to parse all logs produced by the Postfix postscreen daemon. Now, it's able to record the NOQUEUE rejects made by the postscreen daemon into the noqueue SQL table
- updated the trigger_parser() SQL method to handle postfix/postscreen logs
(Trunk r2277):
- Updated ELSE web user interface to introduce new RAWL flag in users managment tabs
- Updated ELSEMC web user interface to allow users to enable or disable the RAWL engine. The new "Enable RAWL" button has been added to the WUI
- Updated various PHP scripts to process the RAWL flag and update the database accordingly
- updated SQL methods and requests for RAWL flag
(Trunk r2276):
- updated the greylse() SQL metohd to implement a new feature requested on the Postfix users mailing list. This feature consists in the following: if a Postfix policy server delegation request comes to a greylse instance (and it must comes from a GENERIC, OUTGOING, MAILING LIST or SENDER ROUTER Postfix server), then the recipient email address may be recorded in the user's permanent senders whitelist. The effect of that is as described there: imagine you've an internal user who send an email to a recipient. Then this recipient will be automatically registered in the user's permanent senders whitelist. So, when the registered recipient, in the futur, will send an email to your internal user (meaning this recipient email address becomes a sender email address), it will be whitelisted (accepted for delivery). This feature can be enabled or disabled in the ELSE and ELSEMC web user interfaces.
(Trunk r2275):
- added new rawl colulmn in SQL tabl users: rawl is for "recipient auto white-list". When this flag is enabled for a user, and each time this user will send emails to recipients, those recipients will be automatically whitelisted as senders. Then if those recipients send email to the user (so here they are senders), the GrayLSE will lt them pass through because they are whitelisted.
(Trunk r2274):
- GreyLSE: make use of GLIBC getnameinfo() function instead of obsolete gethostbyaddr() in GetMyIP() and GetPeerIP() functions
(Trunk r2273):
- I've seen that my own greylse instance, after running for several tens of days, was not able to do its job any more because the SQL requests returned 0 tuples from the database. Possibly, that may be caused by a drop of the SQL connection to the database (or maybe I've restarted my DB daemon...). To know more if this issue will happen again, I've updated the DBHandler() function (this is the Database child of the GreyLSE) in order to log a critical error via syslog with copy of the SQL request in case 0 tuples are returned. Next step will consist in updating this method again to make it re-open the connection to the database server at least once and see if this connection is restored then resume normal operations from the greylse.
(Trunk r2272):
- rewritten to call wui_userslist() SQL method and make use of PDO prepared statement
(Trunk r2271):
- added new wui_userslist() SQL method.
(Trunk r2270):
- rewritten to call wui_usergreylistlist() SQL method and make use of PDO prepared statement
(Trunk r2269):
- added new wui_usergreylistlist() SQL method.
(Trunk r2268):
- rewritten to call wui_buildslist() SQL method and make use of PDO prepared statement
(Trunk r2267):
- added new wui_buildslist() SQL method.
(Trunk r2266):
- rewritten to call wui_greylseclientslist() SQL method and make use of PDO prepared statement
(Trunk r2265):
- added new wui_greylseclientslist() SQL method.
(Trunk r2264):
- rewritten to call wui_greylselist() SQL method and make use of PDO prepared statement
(Trunk r2263):
- added new wui_greylselist() SQL method.
(Trunk r2262):
- rewritten to call wui_pfxerrorslist() SQL method and make use of PDO prepared statement
(Trunk r2261):
- added new wui_pfxerrorslist() SQL method.
(Trunk r2260):
- rewritten to call wui_grulelist() SQL method and make use of PDO prepared statement
(Trunk r2259):
- added new wui_grulelist() SQL method.
(Trunk r2258):
- rewritten to call wui_hrulelist() SQL method and make use of PDO prepared statement
(Trunk r2257):
- added new wui_hrulelist() SQL method.
(Trunk r2256):
- replaced the manual generation of JSON output by json_encode() for SQL results output in some PHP scripts
(Trunk r2255):
- added new wui_brulelist() SQL method.
(Trunk r2254):
- rewritten to call wui_brulelist() SQL method and make use of PDO prepared statement
(Trunk r2253):
- fixed a typo in SendErrMsg() function (removed some ')
(Trunk r2252):
- rewritten to call wui_wrulelist() SQL method and make use of PDO prepared statement
(Trunk r2251):
- start the creation of new SQL methods: all methods name starting by wui_ are meant to replace equivalent PHP scripts. The goal is to update all PHP scripts by removing the most important part of their code and call only one SQL method by PHP script, thus, putting the code logic as much as possible in SQL instead of PHP.
- added new wui_wrulelist() SQL method.
(Trunk r2250):
- Added SendErrMsg() function
(Trunk r2249):
- updated ELSE version number to 0.9.18
(Trunk r2248):
- a messageBox is displayed during the loading of the application, asking the user (only if he is an ELSE admministrator) if he wants to use the ELSE WUI or the ELSEMC WUI.
(Trunk r2247):
- added details about configuration of Apache 2.4 VirtualHost in INSTALL file (authentication on the ELSE database using mod_authn_dbd)
- updated adduser.php, updateuser.php, updatepwd.php to handle user's passwords in apr1md5 apache 2.4 format
(Trunk r2246):
- database schema patch file to go from version 0.9.17 to 0.9.18 of the tool
- added new apr1md5 column to users table. This will contain user passwords in apr1md5 Apache 2.4 format
(Trunk r2245):
- fixed a typo at line 47 of greylse startup script
0.9.17:
-------
Summary:
- ELSE and ELSEMC v 0.9.17 official user's documentation now available in the SourceForge Wiki of the project
- GreyLSE Postfix Policy Server implementing multiple specialized childs (database child, working childs) with pre-forking. GreyLSE provides greylisting with auto-whitelisting, as well as management of site-wide extended filtering and user's sender/recipient white/black listing, on a multi-flow basis (incoming, outgoing, sender routers... postfix servers). It also manages the policies decided on the fly by the ELSE Auto-Cleaner, which is BTW able to become an ACTIVE security feature and not only a PASSIVE one. In addition to that, a new type of list has been implemented: the Holdlist and its associated rules. Holdlist give the possibility to keep emails in the Postfix Hold Queue.
- New tool provided: ELSEMC (ELSE users's personnal Messaging Center). This is a lightweight ELSE WUI useable by any user and allowing him to make searches about emails he sent to somebody, emails he received from somebody, get his reputation statistics, manage his personnal sender and recipient white/blacklist and interact with the GreyLSE to have a view of what is done from the point of view of greylisting. The ELSEMC is bound to the user's email address and all searches are limited to emails containing his email address.
- ELSE WUI: now implements all elements needed to manage GreyLSE server instances and GreyLSE clients
- Implemented SQL table partitioning to enhance performances
- Implemented the ELSEre: ELSE reputation engine. The reputation of every email address, as sender and receiver, is computed. Reputation can go from -100% (worst) to 100% (best) passing through 0 (neutral).
- now, the attachments information are parsed from the email headers and displayed in the search results in the ELSE WUI
- in the ELSE WUI, a new window display all your Postfix servers Panic/Fatal/Warning log lines in real time (for ELSE admins)
Details:
(Trunk r2244):
- bug fix: added email address of the root user in emails table and updated email_id field in users table
- bug fix: added one default server (localhost) in servers table, updated servergroups table accordingly
- without those bug fixes, the ELSE web user interface is not able to be displayed properly right after installation
(Trunk r2243):
- added SQL instruction to allow our maillog database user to use the indexes tablespace, in "CREATE THE DATABASE" section
(Trunk r2242):
- fixed a typo in a "DROP FUNCTION IF EXISTS" statement
(Trunk r2241):
- added versionning information for v0.9.17 in README file
(Trunk r2240):
- bug fixed: replaced roleid by rolename for Role combobox in "Servers" sub-tab under "Manage" tab
(Trunk r2239):
- sub-tabs of the RTAAM panel. For now that's just a copy of other existing panels. All code needs to be done for the RTAAM features implementation
(Trunk r2238):
- added new RTAAM panel to the ELSE WUI, but this panel is disabled for now.
- that's to start to implement the RTAAM engine of the ELSE, which is for Real Time Action and Alerting Module
- all code needs to be done from now
(Trunk r2237):
- fixed a typo in the title bar of the "Auto-Clean BL" panel
(Trunk r2236):
- set the failure flag to false when a greylist rule is updated
(Trunk r2235):
- updated to make use of qmgr_uniqinfos SQL table instead of qmgr table
- updated SQL queries to make use of SQL partitioning context
(Trunk r2234):
- SQL query updated to make use of the referer SQL table
(Trunk r2233):
- changed location of greylse.conf file to /etc/GreyLSE
- moved greylse init script and greylse.conf configuration file in sub-directories
- cosmetic updates to some source files for GNU GPLv3 license
- provided all needed files to allow installation/compilation of the greylse using traditional GNU configure and make tools
(Trunk r2232):
- INSTALL file updated to give more information about the installation of the PostgreSQL database schema
(Trunk r2231):
- added SQL instructions to create the maillog database at the begining of XItool_MailManager_0.9.17.out file
(Trunk r2230):
- XItool_MailManager_0.9.17.out contains the FULL PostgreSQL DB schema (without the SQL FUNCTIONS) for ELSE Version 0.9.17 (DB schema version 958). It can be used to install an ELSE server directly from scratch, and it avoids the deployment of the original database schema (ELSE version 0.6.4) with all successive database upgrade files.
(Trunk r2229):
- GreyLSE: updated for ELSE Database schema v 958 (ELSE v 0.9.17)
(Trunk r2228):
- added SQL updates to set correct values for ELSE v 0.9.17 in versions table
(Trunk r2227):
- droped possibly existing old primary key on emails -> this primary key could be on (email, domain_id) columns
- added new primary key on emails(email)
(Trunk r2226):
- removed a useless raise notice
(Trunk r2225):
- bug fixed: forgot to drop existing primary key before to create the new one for emails table
(Trunk r2224):
- renamed the original XItool_MailManager.out file to XItool_MailManager_0.6.4.out
- updated license text at begining of XItool_MailManager_SQLproc file
(Trunk r2223):
- ELSEMC: when searching using message-id, also search according to rcpt_origto_id column of smtp SQL table with an OR condition with rcpt_to_id column.
(Trunk r2222):
- ELSE Detailed search results window: Added reputation sliders for RFC821 and RFC822 senders, as well as pourcentage numbers
(Trunk r2221):
- ELSEMC: removed the "Delivery Status" header at the top of the grid, in the window displaying search results
(Trunk r2220):
- implemented dynamic forking of policy children
- the greylse should now be able to adapt itself to the load via the forking of policy children. That's controlled in the ELSE interface with Min. childs and Max. childs settings
- every child maintain a busy status in the first 2 bytes of the shared memory segment (a short int)
- this number of busy chilren is used in a comparison with the number of minimum children. if number of busy children is 70% or more of number of forked children + "margin children", the margin children is increased until forked children + margin children reach max. children
(Trunk r2219):
- GreyLSE: added logging to syslog
(Trunk r2218):
- updated the architecture of connection handling: now using pre-forking
- the minchilds setting is now honored (minimum number of childs to pre-fork)
- the maxconchild setting is now honored (number of TCP connections from Postfix to honor)
- the daemon is now able to keep permanently minchilds running: childs are forked consequently
- the maxchilds setting is not honored for now: greylse is ready to open the path to dynamic forking, but code remains to be done
- added new comments and timers in debuging mode
- using CLOCK_REALTIME instead of CLOCK_PROCESS_CPUTIME_ID
(Trunk r2217):
- added minchilds and maxconchild columns to greylses table
(Trunk r2216):
- updated GreylseList model to handle minchilds and maxconchild
- updated Greylse Web User interface editor to include Min childs and Max TCP con./child numeric selectors
(Trunk r2215):
- updated to handle minchilds and maxconchild (number of minimum childs pre-forked, and number of maximum TCP connections that a child will accept before to die)
(Trunk r2214):
- GreyLSE: added new time measure just at the level of the execution of the SQL request
(Trunk r2213):
- bug fix: provided new expandsubdomains(), expandsubdomainsofgroup() and expandsubdomainsofuser() SQL methods that take integers as parameters and return a table of integers instead of bigint. That's something I forgot to fix when users, groups and domainstree tables have been modified to make use of integers instead of bigint, and those SQL methods were not working any more.
- bug fix: fixed fullsearch() SQL method that had mistakes in SQL query used when search had to be performed for users having a list of restricted domains
(Trunk r2212):
- updated trig_smtpd() SQL method to take care of NOQUEUE: hold emails
(Trunk r2211):
- bug fix: forgot to replace BlacklistRulesStore by HoldlistRulesStore when I did the copy of this file
(Trunk r2210):
- updated greylse() SQL method to handle new holdlist rules
(Trunk r2209):
- added new holdlist_rules table. THis table will contain all the rules that will define under which conditions emails will be set on hold (in quarantine, in fact)
(Trunk r2208):
- Provided PHP scripts used to add, update, delete and list holdrules
(Trunk r2207):
- all files containing the definition of Web User Interface objects for HoldList rules features
(Trunk r2206):
- the GreyLSE Postfix Policy Server has been organized a new way from the point of view of its internal architecture: previously, every child processing a postfix delegation policy request created a new connection to the database. That's OK when handling a few tens requests per second (small loads) but it's not enough for ISP type loads. Now, the GreyLSE is organized arround an additional specialized child called the "database child", and "processing childs" that were there before. Any direct SQL interaction has been removed from the processing child. Instead, a processing child makes a request to the database child that will handle database communications. All of that is done using Shared Memory and a set of process Semaphores. Thanks to that, the GreyLSE now use only a unique connection to the database, that is not closed. And it can process far much more Postfix Delegation Request at a time T than before. GreyLSE will not lose CPU time to close and open new database connections for every postfix delegation request.
- also added more comments in DEBUG mode
- in DEBUG mode, processing time in milliseconds is displayed for every processing child
(Trunk r2205):
- GreyLSE: set the type of the NbChilds variable to sig_atomic_t
(Trunk r2204):
- bug fix: the failure column in whiteblacklist_rules table is now set to false when a rule is updated in the Web UI
(Trunk r2203):
- GreyLSE: Avoid creation of useless MCString objects in the Process method
(Trunk r2202):
- ELSEMC: updated to start to provide some statistics about emails sent. Emails sent are computed on GENERIC or OUTGOING server roles. Statistics provided for now include emails successfuly sent, emails bounced and the sum of those two data
(Trunk r2201):
- updated SQL requests executed in fullsearch() SQL method to avoid possible wrong results (when using Postfix having QIDs that can repeat themselves): the date filter on qids was not used.
- removed the lock table in smtp2parse SQL method, because that introduces too much problems than solutions
(Trunk r2200):
- the GreyLSE should now (more) correctly count the number of childs forked, and honor the maxchilds setting.
(Trunk r2199):
- bug fix: a search by message_id in the ELSEMC could work or not. That was due to the fact we convert the message_id entered by the user in the Web UI in lower case, but the corresponding message_id field in the SQL table and SQL search request wasn't converted in lowercase itself
(Trunk r2198):
- count number of childs created
- using SIGCHLD to decrease childs when they die
- Using current number of running childs to enforce maxchilds database setting for the greylse daemon
- If number of childs running at maximum, directly close the TCP connection with Postfix: that should let Postfix send a policy delegation request later. This is expected to be useful to manage GreyLSE overloads because of extremely high email throughput, causing too much policy delegation requests sent to the GreyLSE
(Trunk r2197):
- due to deadlocks issues when tasks_5min() SQL method is executed with huge email trafic, reduced the number of processed rows to 5000 instead of 10000 in qmgr2parse() and smtp2parse() SQL methods that are called by tasks_5min(), and also lock servers table with exclusive mode in smtp2parse() SQL method. That should avoid accumulation of data in the corresponding tables and also that should avoid the cancel of tasks_5min() execution by the database backend, as well as the fact this SQL method used more and more time just to try to be executed.
(Trunk r2196):
- GreyLSE: in Process() method, open the connection to the database just when necessary, and not at the very begining of the method where it was for nothing
(Trunk r2195):
- updated README and LICENSE to GNU GPLv3
(Trunk r2194):
- Updated README and LICENSE file with GNU GPLv3
(Trunk r2193):
- the greylse() SQL method has been updated to take care of the situations where postfix servers are members of several different groups. parsing of white/grey/black list rules deduct the server group from the server id now.
(Trunk r2192):
- bug fix: in table greylist, a constraint defined on column srvgroup_id was referencing a wrong table/column (column sg_id of table servergroups). This is now referencing the right table/column (column group_id of table groups)
(Trunk r2191):
- added a 'Failed' column to white/grey/black lists rules. This column will have a checkbox if the corresponding rule generated an exception when evaluated by the greylse daemon
(Trunk r2190):
- the data provided for white, black and greylist rules now contain the failure flag
(Trunk r2189):
- added failure boolean column to GreylistRulesList, WhitelistRulesList and BlacklistRulesList models
(Trunk r2188):
- implemented processing of global whitelist and blacklist rules in greylse() SQL method
(Trunk r2187):
- added failure column in table whiteblacklist_rules
(Trunk r2186):
- updated SQL request to make it a little bit faster (make use of date column of qids table)
(Trunk r2185):
- updated to generate the greylse version of the white/black list rule entered by the user
(Trunk r2184):
- Added AutoCleanReport() SQL method
- started to implement greylisting rules in greylse() method
(Trunk r2183):
- updated to use the new AutoCleanReport() SQL method
(Trunk r2182):
- added a failure column to greylist_rules table. This flag can be true for any rule that generate a SQL exception
(Trunk r2181):
- updated to generate the greylse version of the greylist rule entered by the user
(Trunk r2180):
- updated default value of column disable in table greylse_clients to true, to avoid newly installed greylse instances to perform their job immediately. An action of an ELSE admin is now necessary to explicitly enable clients of greylse instances
(Trunk r2179):
- Updated the SQL request to return the list of greylse instances even if there are no clients associated to them
(Trunk r2178):
- the "whitelist rules" and "blacklist rules" sub-tabs under the "WGB lists" tab are done, on the same basis of the "greylist rules" sub-tab committed previously.
- provided models and stores for whitelist and blacklist data storage
- provided PHP scripts to add, update, delete white/black list rules as well as list them in the grids
- However at this time, the GreyLSE daemon does not honor the white/black list rules: this part remains to do
(Trunk r2177):
- completed the function onGRuleAdd()
(Trunk r2176):
- bug fix: the JSON variable was incorrectly named (servers instead of rules)
(Trunk r2175):
- added whiteblacklist_rules SQL table to manage global white and black lists
(Trunk r2174):
- the "greylist rules" sub-tab under the "WGB lists" tab should be completely operational now.
- model and store used to store greylist rules have been updated for the proxy to be removed from the store and moved to the model
- provided PHP scripts to add, update, delete greylist rules as well as list them in the grid
- However at this time, the GreyLSE daemon does not honor the greylist rules: this part remains to do
(Trunk r2173):
- added column matched to greylist_rules table
(Trunk r2172):
- Added new "Greylist Rules" sub-tab under the "WGB Lists" tab. This is where users will be able to define rules to control greylisting feature of the GreyLSE instances.
- added corresponding model and store to manage greylist rules
- added the UI dialog to handle addition of greylist rules
- That's only the first coding of this feature. For now, there is still work needed and it's not operational.
(Trunk r2171):
- created greylist_rules table. It will contain the rules controlling greylist behavior applicable by the GreyLSE daemon
(Trunk r2170):
- bug fix: the last delete operation could send a failure JSON answer when it wasn't the case
(Trunk r2169):
- Added Stop and Start Greylse instance buttons in sub-tab "GreyLSEs" of tab "Manage"
(Trunk r2168):
- Added stop and play icons, updated lse.css file
(Trunk r2167):
- added 2 fields in the GreyLSE edit window for greylse service start and stop commands
- updated GreylseList model and PHP scripts accordingly
(Trunk r2166):
- added columns startcmd and stopcmd to greylses table
(Trunk r2165):
- adjusted some column widths in "Manage" tab panel, "users" and "billing reports" sub-tabs
(Trunk r2164):
- renamed tab "Log re-Builder" as "Logs" in the ELSE WUI
(Trunk r2163):
- Bug fix: in the "ACLs" tab, when editing policy objects, a wrong PHP instruction used to execute a SQL query prevented to delete any object in the policy
(Trunk r2162):
- added the anti-spam icon in the tab title "WGB Lists"
(Trunk r2161):
- removed the tab "Import Message-ids" from the main window
- added an "import" button in the "Search by Message-ID" panel: the tab "Import Message-ids" has been moved there
(Trunk r2160):
- bug fix: a root user was not allowed to view the ACL policies if the group he is member of had the write flag set to false and if the policy in question was created by somebody in the same group
(Trunk r2159):
- ELSE bug fix: clicking the save button when editing a billing report is now working.
(Trunk r2158):
- added all ExtJS and PHP scripts to manage GreyLSE client settings
- updated "GreyLSEs" management panel accordingly
(Trunk r2157):
- the update of all configuration settings of a greylse instance is now possible
(Trunk r2156):
- provided the deletegreylse.php script that is used to delete a GreyLSE instance when the user click on the delete button in the "GreyLSEs" tab of the Management panel
(Trunk r2155):
- when double-clicking on a Greylse in the Greylse management UI, a new panel appears to let the user configure the greylse instance
- updated GreylseList model and greylselist.php
(Trunk r2154):
- there is a new "GreyLSEs" tab under the ELSE "Manage" tab: this will be used to display and manage the GreyLSE daemons, handling greylisting, blacklist and whitelist features
(Trunk r2153):
- The content of the User's GreyList can now be downloaded as Excel files, in the ELSE Messaging Center
(Trunk r2152):
- The content of the Sender blacklist, Sender Whitelist and Recipient Whitelist can now be downloaded as Excel files by the ELSE Messaging Center users
(Trunk r2151):
- Added new column disable_greylisting to table greylse_clients
- Updated greylse() SQL method to honor disable and disable_greylisting of greylse_clients table: disable flag completely disable the grelyse features, and disable_greylisting flag disable only greylisting feature but perform all whitelists/blacklists features
(Trunk r2150):
- provided all GUI parts and scripts to make the new "Postfix warning/fatal/panic messages" window working
(Trunk r2149):
- updated trig_error() SQL method to keep only the relevant part of the log line
(Trunk r2148):
- updated trigger_parser() SQL method to count number of log lines processed when processing warning/panic/fatal
- make use of trim() in trig_error() SQL method
(Trunk r2147):
- create errors table, that will record all panic/fatal/warning postfix log lines
- updated timestamp_modulo() SQL method to be written in SQL language instead of plpgsql: we win some tens of milliseonds for each call
- added trig_error() SQL method that will register any postfix panic/fatal/warning in errors SQL table
- updated trigger_parser() SQL method to call trig_error() SQL method when necessary
(Trunk r2146):
- ELSE Main Window is now closable. Close the window is the LOGOUT action.
(Trunk r2145):
- Added a logout button to the ELSEMC
(Trunk r2144):
- Added the logout icon
(Trunk r2143):
- number of received emails is also computed if email server is GENERIC, and not only INCOMING as previously
(Trunk r2142):
- ELSEMC: Added version number, licence and author
(Trunk r2141):
- Added a "Change Password" button in the ELSE Messaging Center
- Added ELSEMCPwd.js for the Change Password dialog box
(Trunk r2140):
- Added new "greylist" button in the ELSE Messaging Center. This button displays a grid showing the user what are the emails sent to him currently greylisted, and sent only once. That will allow the user to add the email addresses in his sender whitelist by clicking on the "+" icon.
- added all javascript files and PHP files to manage this new "greylist" feature
(Trunk r2139):
- in checkemail() and getemailid() SQL methods, make use of lower function to avoid recording email addresses with upper case letters in the database: that could cause some issues with the GREYLSE and the ELSEMC
(Trunk r2138):
- updated bill_r_stats() SQL method to also compute reputation for recipients given by rcpt_origto_id column, and not only from rcpt_to_id column as before
- added an index on greylist(rcptto_id)
(Trunk r2137):
- updated some SQL requests to return results also based on rcpt_origto_id column of smtp table, in addition to the rcpt_to_id column
(Trunk r2136):
- set mimetype of table mimetypes to varchar(128)
(Trunk r2135):
- bug fix in greylse() SQL method about autowldateend
- added some "drop function" commands needed for schema update
- bug fixes in XItool_MailManager_0.9.16_0.9.17.out
(Trunk r2134):
- added daemon column to greylses SQL table
(Trunk r2133):
- removed the DAEMON configuration setting from the greylse.conf configuration file, as it is now directly configured in the database
- updated GreyLSE.C to read from the database if it must execute as a daemon or not
(Trunk r2132):
- the greylse() SQL method now enforce email addresses blacklisted by the ELSE autocleaner, as well as users personnal sender blacklist, sender whitelist and recipient whitelist from the ELSEMC (ELSE Messaging Console)
(Trunk r2131):
- removed vacuum operation from contab (we let the autovacuum doing the job)
- added a job for select es_helo() in crontab
(Trunk r2130):
- provided RedHat startup script for the GreyLSE (to be copied in /etc/init.d)
(Trunk r2129):
- GreyLSE can now run as a daemon
(Trunk r2128):
- Now the GreyLSE do the setgid and setuid
(Trunk r2127):
- GreyLSE: Updated to open a new database connection each time a child is forked
(Trunk r2126):
- updated greylses SQL table with those additional columns: firsthit_validity, gl_msg, gl_policy, gl_smtpcode
- updated greylse() SQL method to apply greylisting policy according settings of the greylses table
(Trunk r2125):
- removed index on greylist_nets(network)
- removed constraint on greylist(network)
- when checking hostname/IP pair in greylse() SQL method, doing it with checkhostbyip() instead of checkhost()
(Trunk r2124):
- created greylist_nets SQL table. this table will handle hosts to networks pairs: network is computed with netmask /24 for IPV4 and /120 for IPV6 applied to host IP address. BTW, the GREYLSE will apply greylisting feature on the network, not on the IP address, to handle emails that are re-sent from different server IPs each time.
- added checkhostnet() SQL method
- implemented first greylisting SQL algorithm in greylse() SQL method
(Trunk r2123):
- updated some PHP scripts to take care of email_id field
(Trunk r2122):
- Added an E-Mail field in the UserAdd and UserEdit dialogs, updated UsersList model for that
(Trunk r2121):
- added email size and number of attachments columns in the Excel Export
(Trunk r2120):
- added greylist SQL table
(Trunk r2119):
- GreyLSE: Bug fix: added a space in the returned answer to Postfix in some conditions (after the action).
(Trunk r2118):
- Added the 'Recipient Server' column to the results window of the ELSEMC
- Added rcpthost field to model ELSEMCSearch
- Updated ELSEMCsearch.php to provide the recipient server in the results provided by SQL requests
(Trunk r2117):
- updated the SQL requests in ELSEMCsearch.php to execute searches with case insensitive in ELSEMC
(Trunk r2116):
- added a waitBox when loading results of a search in the ELSEMC
- replaced 'sent' by 'delivered' in the results of a search in the ELSEMC. This is probably more explicit for most of the users (that are not SMTP admins)
(Trunk r2115):
- enabled stateful provider: Window and Grid settings of ELSE Messaging Center results are kept in the user's browser cookies
(Trunk r2114):
- Made the Messaging Center Results Window resizable
(Trunk r2113):
- added nb_attachments field to ExtendedSearch model
- Added Attachments column to grid panel showing extended search results
- Updated dofullsearch.php script to return nb_attachments column in results
(Trunk r2112):
- updated fullsearch() SQL method to return nb_attachments column in the results
(Trunk r2111):
- added new column nb_attachments to qmgr_uniqinfos table
- updated removeattachmentswaitarea() SQL method to count number of attachments added to the attachments table, and update the qmgr_uniqinfos table
(Trunk r2110):
- ELSEMC: for the search on senders, also perform the search on RFC822 senders
(Trunk r2109):
- updated to handle email size in extended search results: added size field in ExtendedSearch model, updated MyFullSearchWindow view to add related column, updated SQL request made in dofullsearch.php
(Trunk r2108):
- updated fullsearch() SQL method to provide email size in the returned results
(Trunk r2107):
- added table userswgb. This table will contain user's personnal black and white lists that they can manage using the ELSE MESSAGING CENTER (ELSEMC)
(Trunk r2106):
- here are the files for the ELSE MESSAGE CENTER component
- This component is a personnal user's messaging center allowing ELSE users having new "mconly" permission to use it
- It gives statistics about daily delivery, indication of user's reputation as both a sender and a recipient
- It allows the user to search for an email he sent, or an email he received (or was supposed to have sent or received)
- Also, users can configure their own Sender Blacklist, Sender Whitelist, and Recipient Whitelist
(Trunk r2105):
- Checking users permissions in app.js, and displays messagebox accordingly.
- Viewport is not created any more if user account is not existing or disabled
(Trunk r2104):
- bug fix: when disabled flag is set to true, all other permission flags are kept to false.
(Trunk r2103):
- reworked SQL request to check user's permissions
- added a "disabled" account flag
(Trunk r2102):
- bug fix: in removendrwaitarea() SQL method. the update command used a wrong column name, and no NDR was inserted in the database
- in fullsearch() SQL method, added new tests with t4.date column to benefit of additional perfs in context of table partitioning
(Trunk r2101):
- added model and store to manage email attachments
- updated simple search results window to show attachment details in a UI section below the UI header section
- started to rework some PHP scripts (recipientslist.php, dosearch.php, qidslist.php, attachmentslist.php) to take benefit of database table partitioning
- in simple search results window, added reputation system indicators for the RFC821/RFC822 email sender addresses
(Trunk r2100):
- still more updates for table partitioning
- added another getreputation() SQL method that provide email address reputations based on email_id
- added another qidslist() SQL method, that provides the date information in returned data (used in the context of table partitioning)
(Trunk r2099):
- Added a resolution of 1 week for some graphs
(Trunk r2098):
- added new getqid() SQL methods
- reworked trig_qmgr() SQL method
- replaced checkqid() calls by getqid() calls where possible: this method is much faster than checkqid() call
- dropped qmgr_wait table, now useless thanks to the reworking of trig_qmgr() SQL method. dropped SQL methods that were used for qmgr_wait table maintenance
(Trunk r2097):
- a lot of changes everywhere still to make wide use of table partitionning, also created some new qmgr_wait and qmgr_cache tables
- creation of reputations table that manage email addresses reputations from a sender or receiver side point of view, updated bill_s* and bill_r SQL methods to compute reputations, created getreputation() SQL method. In short, reputation of any email address goes from -1.0 (bad) to 1.0 (good) passing by 0 (neutral). Reputations are updated differently every time an email is rejected, bounced, deferred or sent for sender and recipients. Reputations is at this time a new indicator taht greatly help to discover attacking sender email addresses or recipients email addresses having issues, for instance. All of this is called the ELSERE (ELSE Reputation Engine)
(Trunk r2096):
- updated some SQL requests in dosearch.php and drawmailflow.php to benefit of table partitioning DB structure
(Trunk r2095):
- added smtp_partition() SQL method with its associated trigger to start to test table partitioning in order to enhance performances. This is done for smtp table. A referenced column constraint had to be removed also, otherwise no partitioning for this table would have been possible
- updated SQL code of some *_partition() SQL methods to generate better indexes
- optimized some SQL requests in the context of table partitioning (like a part of fullsearch() SQL method for example)
(Trunk r2094):
- created qmgr_wait table to handle QMGR log lines when they are sent to the database in a wrong order. log lines are then processed when enough of them have been sent and the right order is there again. This table works in conjonction with the qmgr_cache table and the processing of the cleanup log lines, and it is needed in the context of table partitioning
- updated trig_qmgr() SQL method to handle qmgr_wait table
- added qmgr_partition() SQL method with its associated trigger to start to test table partitioning in order to enhance performances. This is done for qmgr table
(Trunk r2093):
- fixed a mistake in the SQL command
(Trunk r2092):
- create new qmgr_cache table. It will contain for all running email transactions at QMGR stage level, the qid_id and the date of the first transaction. This cache table shoud be maintained by the trig_cleanup() SQL method. The goal of this table is to drastically accelerate searches in postfix related SQL tables in the context of table partitioning.
- added new dateend column to qmgr_uniqinfos table. This column will contain at the very start of an email transaction, the date of this transaction plus a margin of 5 days, and it will contain the date of the end of the transaction at QMGR stage level when the email transaction is removed from QMGR pipeline. The goal of this column is to drastically accelerate searches (especially the extended search in the Web User Interface) in the context of table partitioning.
- all postfix related parsing SQL methods like trig_smtp(), trig_smtpd(), trig_qmgr(), ... have been updated/rewritten to act much better in the context of table partitioning. SQL requests used in those SQL methods have been reworked and optimized.
- did a lot of testing to redesign the indexes in the context of table partitioning, and dropped index used until now then recreated new indexes that should allow better performances in the context of table partitioning
- other updates made in the context of table partitioning
(Trunk r2091):
- changed type of event column of table es_send_receive to be "char" instead of char(1), in order to benefit of plain storage for this column instead of extended
(Trunk r2090):
- added smtpd_partition() SQL method with its associated trigger to start to test table partitioning in order to enhance performances. This is done for smtpd table
(Trunk r2089):
- added qmgr_uniqinfos_partition() SQL method with its associated trigger to start to test table partitioning in order to enhance performances. This is done for qmgr_uniqinfos table
(Trunk r2088):
- added cleanup_head_partition() SQL method with its associated trigger to start to test table partitioning in order to enhance performances. This is done for cleanup_head table
- also in previous commit, added billing_stats_partition() SQL method with its associated trigger
- added vacuum_stats() SQL method, in case PostgreSQL ELSE Database admins would need it
(Trunk r2087):
- added cleanup_partition() SQL method with its associated trigger to start to test table partitioning in order to enhance performances. This is done for cleanup table
(Trunk r2086):
- added es_msgtrack_partition() SQL method with its associated trigger to start to test table partitioning in order to enhance performances. This is done for es_msgtrack table
- added billing_stats_partition() SQL method with its associated trigger to start to test table partitioning in order to enhance performances. This is done for billing_stats table
(Trunk r2085):
- created new index on iis_cookie_pairs(value) to considerably speed up execution of clean_es_ews_wait() SQL method (executed every 5 minutes by cron)
- added es_ews_partition() SQL method with its associated trigger to start to test table partitioning in order to enhance performances. This is done for es_ews table
- updared clean_es_ews_wait() and trig_es_ews() SQL methods to avoid permanent delete and insert of same data
(Trunk r2084):
- added es_send_receive_partition() SQL method with its associated trigger to start to test table partitioning in order to enhance performances. This is done for es_send_receive table.
- rewritten es_helo() SQL method to benefit of partitioned table
- continue to update DB schema for partitioned tables in XItool_MailManager_0.9.16_0.9.17.out
(Trunk r2083):
- removed specific cleaning of header table from the cleanner() SQL method, because maintenance is done by another method for partitionned tables
(Trunk r2082):
- as header and rawlogs tables are now partitionned, drop unecessary indexes and constraints on parent tables
- removed specific cleaning of header table from the cleanner() SQL method, because maintenance is done by another method for partitionned tables
(Trunk r2081):
- SQL crontab: added a line to execute partition_cleaner() SQL method every day at midnight
(Trunk r2080):
- added 'lock table' instructions on rawlogs and servers tables, in rawlogs_partition() SQL method
- updated any 'insert into logs' instructions to handle ref_id column
- created parition_cleaner() and delete_partition() SQL methods to handle automatic DROP of partitioned tables after the configured number of days
(Trunk r2079):
- bug fix: in timestamp_modulo() SQL method. Added a trunc(). Without that, when we had a date for example of '2013-12-01 23:59:59.568952', the method returned '2013-12-02 00:00:00', which generated inertion errors in new partitionned tables based on date, at the level of the check constraints.
(Trunk r2078):
- added header_partition() SQL method with its associated trigger to start to test table partitionning in order to enhance performances. This is done for header table.
(Trunk r2077):
- set owner of indexes tablespace to maillog;
- SQL: created the else schema
- added rawlogs_partition() SQL method with its associated trigger to start to test table partitionning in order to enhance performances. This is done for rawlogs table.
(Trunk r2076):
- updated a lot of check*() SQL methods to implement unique_violation exception management
(Trunk r2075):
- Added info16x16.png and search16x16.png icons
(Trunk r2074):
- Added icons in the Info and Search tabs
(Trunk r2073):
- added additional tables to split Exchange server logs in. Updated es_send_receive table consequently, as well as various SQL methods
(Trunk r2072):
- added internal message id, network message id fields when displaying information on an email handled by Exchange server 2010/2013. Updated corresponding SQL request in dosearch.php
(Trunk r2071):
- EXCHANGE_SRV_LOGS file updated for parsing of Exchange 2013 Server Message tracking logs
(Trunk r2070):
- provided getNbCores.sh script for monitoring purposes
(Trunk r2069):
- provide snmpd.conf template file for Postfix management server
(Trunk r2068):
- provided cacti stuff to monitor individual postfix servers as well as postfix management server (ELSE server). This is to launch the iostat command every 5 minutes on the Postfix management server
(Trunk r2067):
- provided cacti stuff to monitor individual postfix servers as well as postfix management server (ELSE server)
(Trunk r2066):
- updated PostgreSQL user crontab
(Trunk r2065):
- added several new tables to reorganize tables used to store Microsoft Exchange Server data
- updated original tables used to store Microsoft Exchange Server data to make use of new tables
- those tables have been reorganized to avoid duplication of the same data for various table fields
(Trunk r2064):
- bug fix: in common.php: use new ref_id column in logs table
(Trunk r2063):
- Database Schema updated for ELSE v 0.9.17, DB rev. 958
(Trunk r2062):
- added referers SQL table and checkreferer() SQL method
- fixed a bug in checkuseragent() SQL method: at the end of the funciton, RETURN had a wrong variable name
(Trunk r2061):
- added GreylseList model and GreylseListStore store
(Trunk r2059, r2060):
- resized antispam.png icon
(Trunk r2058):
- added antispam.png file and updated CSS file
(Trunk r2057):
- logs.php: updated to use useragents table in SQL requests
(Trunk r2056):
- using coalesce() SQL function where possible
(Trunk r2055):
- some files updated for version 0.9.17
(Trunk r2054):
- updated all DBLog* functions because of logs SQL table structure change
(Trunk r2053):
- database schema patch file to go from version 0.9.16 to 0.9.17 of the tool
*** PLEASE NOTE THIS PATCH FILE MIGHT NEED A LOT OF TIME TO BE APPLIED TO YOUR DATABASE IF YOU PROCESS EXCHANGE SERVERS LOGS, DUE TO THE FACT DB TABLES FOR EXCHANGE LOGS ARE RE-ORGANIZED IN THEIR STRUCTURES. EXCLUSIVE LOCKS ARE MADE ON THE TABLES. ***
- added new ms_orgs and useragents tables.
- updated structure of logs, iis_advanced, iis_w3svc, es_ews, es_rca tables to use ms_orgs and useragents tables. That will save space in the first tables because a lot of redondant information will now be stored in the last tables.
- in consequence, updated all SQL methods that are using those tables
- added checkmsorg() and checkuseragent() SQL methods
0.9.16:
-------
Summary:
- ELSE v 0.9.16 official user's documentation now available in the SourceForge Wiki of the project
- Introduction of the GreyLSE, a C/C++ implementation of a Postfix Policy Server with core logic in SQL methods. This is a big new feature of this ELSE version.
- New Auto-cleaner feature with reporting and automated blacklisting. This is a big new feature of this ELSE version.
- Postgrey logs are now parsed by the ELSE
- Content of 'My E-mail Search Requests' can now be cleared
- Excel files are not generated on client side any more, but on server side thanks to the use of a new library
- Postfix queues can now be cleaned: the 'Delete' button of the window showing emails in queues have some code insisde (SNMP needed)
- Postfix queues can now be flushed (SNMP needed)
- use of the JQuery library
- started to update standard data grids to buffered data grid to get big performance improvements in Web User Interface
- reviewed log Rebuilder features: they are now done entirely by SQL methods
- SQL Performances improvements
Details:
(Trunk r2052):
- added versionning information for v0.9.16 in README file
(Trunk r2051):
- updated logs.php script to make strings conform to HTML
- updated ShowLog.js to use a html editor instead of a textarea for the message property
(Trunk r2050):
- logrebuilder.php has been reworked to use the rebuildlogs() SQL method instead of doing all SQL requests in the PHP script itself. Performance is better
(Trunk r2049):
- revised rebuildlogs() SQL method
(Trunk r2048):
- added rebuildlogs() SQL method. We have inside what was previously done by the logrebuilder.php script
(Trunk r2047):
- added user group and server role drop down lists and removed server field in log rebuilder tab, for Rebuild Logs
(Trunk r2046):
- changed type of reference column to text in es_msgtrack table
(Trunk r2045):
- simplified parsing of QIDs in all SQL methods
(Trunk r2044):
- bug fix: in trig_bounce() SQL method, an issue in the parsing of the infos field prevented the captured NDRs to be registered correctly in the database
(Trunk r2043):
- data grid for the "Auto-clean BL" tab is now using a buffered store to enhance performance
(Trunk r2042):
- bug fix in some *.php files: escaped the rcptto field in returned data, to prevent issues when displaying data in the WUI
(Trunk r2041):
- The ELSE logs available in the data grid under the "Logs" sub-tab, "Manage" tab has been turned in a buffered grid. Paging toolbar has then been removed
(Trunk r2040):
- replaced column server varchar(?) by column server_id int4 in table es_ews_wait
- updated trig_es_ews() and clean_ew_ews_wait() SQL functions accordingly
(Trunk r2039):
- bug fix in some *.php files: escaped the rcptto field in returned data, to prevent issues when displaying data in the WUI
(Trunk r2038):
- bug fix in some *.php files: escaped the rcptto field in returned data, to prevent issues when displaying data in the WUI
(Trunk r2037):
- bug fix: removed typo issues in PolicyCheck.js (commas)
(Trunk r2036):
- Bug fix: typo fix in WGBLists.js (comma)
- Update MyContainer.js to check user's permissions each time a tab is changed in main window, so that some tabs can be hidden or showed
(Trunk r2035):
- bug fix: checkperms.php: $cleaner variable was not initialized
(Trunk r2034):
- Bug fix: typo fix in MyACBLPanel.js that prevented ELSE to start
- updated etc/config.php for ELSE v 0.9.16
- Splash loading screen is now deported in index.html file. Removed impacted styles from lse.css file, and updated app.js file consequently
(Trunk r2033):
- fixed some typo issues
(Trunk r2032):
- removed all deprecated SQL methods
(Trunk r2031):
- enabled cleaning of Exchange QIDs registered in pfx_cache table in SQL method trig_es_trck()
(Trunk r2030):
- altered type of server column in pfx_cache and servers tables to be varchar(64)
(Trunk r2029):
- in the Show Server Queue window, added a "Flush Queue" button
- added flushq.php script, that makes a SNMP call to a postfix server to launch postqueue -f command
- added flush icon
(Trunk r2028):
- README file: Added link to the SourceForge project page Wiki to README file
(Trunk r2027):
- added an index on header(date)
- added cleanning of header table in cleanner() SQL method
- Bug fix: email addresses that are blacklisted by the auto-cleaner were counted in the statistics even if they were already blacklisted. This is now fixed
(Trunk r2026):
- Added cleanning of billing_stats table in cleaner() SQL method
(Trunk r2025):
- updated fill color of columns to be light grey, in Auto-clean report graph window
(Trunk r2024):
- added an index on autoclean_reports(smtp_id) to speedup execution time of cleanner() SQL method
(Trunk r2023):
- it's now possible to manage the content of the auto-clean blacklist in the "WGB lists" tab, "Auto-clean BL" sub-tab. Email addresses in this blacklist can be enabled, disabled or removed. The content of this blacklist can also be downloaded as an Excel file.
(Trunk r2022):
- Added enable and disable icons
(Trunk r2021):
- provided delsenderfromq.php script, used by the Delete button of the Show Queue window. This script takes the server name as well as an email address as parameter, and do a SNMP call on the server to delete all emails from the sender in the postfix queue
(Trunk r2020):
- in the window showing emails in server queues, the Delete button is now working. This button is to delete all emails of a particular sender on the considered server. It calls delsenderfromq.php script, doing itself a SNMP call on the selected server
(Trunk r2019):
- provided postfix-delete.pl script that will be used by the ELSE to remove emails from postfix deferred queues
(Trunk r2018):
- removed the ux.exporter library that is now useless and replaced by server side generated files to be downloaded using JQuery and PHPExcel
(Trunk r2017):
- updated export feature of Billing reports window to make use of server-side genrated excel files
(Trunk r2016):
- updated export feature of Reporting and statistics tab of main window to make use of server-side genrated excel files
(Trunk r2015):
- updated export feature of Auto-Cleaner statistics window to make use of server-side genrated excel files
(Trunk r2014):
- Fix a type: wrote httpMthod instead of httpMethod in MyFullSearchWindow.js and MyNoQueueWindow.js
(Trunk r2013):
- updated export feature of No Queue Search Results window to make use of server-side genrated excel files
(Trunk r2012):
- updated export feature of Extended Search Results window to make use of server-side genrated excel files
(Trunk r2011):
- updated export feature of auto-cleaner windows to make use of server-side genrated excel files
(Trunk r2010):
- provide third party component: PHPExcel (http://phpexcel.codeplex.com/)
- provide third party component: JQuery (http://www.jquery.com)
- provide third party component: jquery.fileDownload.js (http://jqueryfiledownload.apphb.com/)
- updated INSTALL file containing sections for new third party components used. they are now dependencies of the tool
- those new libraries are used to provide server-side generated Excel files used by the export features of the tool, in replacement of the client-side generated from grids CSV files, managed by the old downloadify library
- the tool will see all its download buttons updated to use the new way of exporting the data, and once done, the ux.exporter and downloadify and all their dependencies will be removed. The new solution provided there is more flexible at several levels.
- first feature of the tool that is updated is the download button in the Queue List window. For this the exportqueuelist.php script is provided
- updated index.html file to include the new third parties libraries
(Trunk r2009):
- added download icon
(Trunk r2008):
- Removed the 'download' button using client-side generated *.CSV file
- Added 'download' button using server-side generated *.XLSX file
(Trunk r2007):
- Constrain main window to the browser display area
(Trunk r2006):
- created a set of new covering indexes on some key tables (cleanup, cleanup_head, qmgr_uniqinfos, smtp, smtpd) in order to drastically improve response time of SQL requests and stored methods used in the ELSE
- Updated and optimized fullsearch() SQL method to make better use of newly created covering indexes to drastically improve response time using the ELSE extended search feature, statistics, ...
(Trunk r2005):
- added cleaning of smtp, smtpd, qmgr, qmgr_uniqinfos, cleanup_head tables in cleanner() SQL method. Even if those tables are already cleaned by all the ON DELETE CASCADE that are everywhere, because the simple fact QIDs are not unique in Postfix <= 2.8 makes that some very old data may stay in those tables
(Trunk r2004):
- created new indexes on es_msgtrack and es_send_receive tables, removed some old ones. The new indexes drastically enhance performances of request done in es_helo() SQL function. The gain is important as this SQL method is a maintenance one called every 5 minutes by crontab
(Trunk r2003):
- bug fix: sub-tabs in tab 'Manage' were not displayed any more because of a wrong variable name (abstractcomponent instead of component)
(Trunk r2002):
- added insertedon column to autoclean_bl table: it will record the timestamp at which the data have been inserted in this table
(Trunk r2001):
- Solved a Web UI design issue at the level of the 'Manage' Tab of the main window: replaced ManageTabPanel.js file by Manage.js file
- Removed fixed height in script code for all sub-tabs of the 'Manage' tab, and added autoScroll
(Trunk r2000):
- In the main window, renamed 'Auto-cleaner' Tab as 'WGB Lists'. Meaning White/Grey/Black Lists. This tab will include sub-tabs to manage auto-cleaner and all System Lists
- Move Auto-cleaner tab under new 'WGB Lists'
(Trunk r1999):
- added a loading mask when displaying auto-clean report statistics
(Trunk r1998):
- For auto-clean distinct recipients list, added also the domain name in the grid, and grouping by domain name
- added icons in the Web UI
(Trunk r1997):
- added column 'enabled' to SQL table autoclean_bl
(Trunk r1996):
- bug fix: a dataindex was wrongly named, to the blacklisted column of the grid displayed empty values
(Trunk r1995):
- Added the graph Web UI to the auto-clean report window. THis shows the number of times the auto-cleaner is triggered, and the number of times the auto-blacklist is activated
(Trunk r1994):
- Added two small icons (grid and chart)
(Trunk r1993):
- added generateAutoCleanReport() SQL method that provide statistics to make the graphs for auto-clean
(Trunk r1992):
- added chart16x16 icon declaration
(Trunk r1991):
- added autoclean_stats SQL table. It is used to maintain real time statistics about number of times per slice of 5 minutes interval the auto-cleaner is triggered
- updated the trig_smtp() SQL function to generate statistics for the auto-cleaner
(Trunk r1990):
- When the auto-cleaner reporting window is displayed, make the content of the grid to be loaded on the present day by default
(Trunk r1989):
- Added a "Distinct Recipients" button in the Auto-clean reporting window. This one as it is named, will dipslay the list of distinct recipients.
- when double-clicking a recipient line in the auto-clean report window, it now displays the full details of the email transaciton (the search results window in fact)
(Trunk r1988):
- extended width of the Entry ID Combobox to 375 px
(Trunk r1987):
- auto-cleaner: Added grouping by messageid field
- auto-cleaner: Added clienthost, clientip and dsn columns in the grid
- auto-cleaner: Added renderer for status field
(Trunk r1986):
- added clienthost, clientip and dsn fields to CleanReportList data model
(Trunk r1985):
- Continue to code the Web UI for showing the Auto-Cleanner reports
(Trunk r1984):
- start implementation of auto-clean reporting Web interface
(Trunk r1983):
- removed ui directory
(Trunk r1982):
- removed the ui directory, and moved all files inside at the base of the view directory. Updated all file headers consequently.
- added constrainHeader: true directive for all windows where that is relevant
- some minor bug fixes in *.js files
- cosmetic minor changes
(Trunk r1981):
- removed the ui directory, and moved all files inside at the base of the view directory. Updated all file headers consequently.
- added constrainHeader: true directive for all windows where that is relevant
- some minor bug fixes in *.js files
- cosmetic minor changes
(Trunk r1980):
- removed the ui directory, and moved all files inside at the base of the view directory. Updated all file headers consequently.
- added constrainHeader: true directive for all windows where that is relevant
- some minor bug fixes in *.js files
- cosmetic minor changes
(Trunk r1979):
- implementing the new AUTO-CLEANNER feature of the ELSE
- a new tab has been added in the main interface with name "Auto-cleanner". It is enabled for ELSE administrators and users with write privileges
- this new feature shows a new grid in which auto-clean policies can be registered. Basically, each policy defines a sender email address to be monitored on servers having a specific role, and that are members of a specific group. Status of all emails sent to recipients by this sender is then monitored on the fly. If emails sent to any recipient are bounced, or deferred for more than 2 days, then the email transactions for such recipients are registered in the reporting SQL tables and also possibly in the auto-clean blacklist
- the GreyLSE daemon of the ELSE (which is my implementation of a policy server for Postfix with logic all done in the database by the mean of stored SQL methods), can then use this auto-maintained blacklist to not send any email to those recipients. The goal is I think interesting: take for example the case of a huge mailing-list of several hundreds of thousands of recipients. For sure, this mailing list will contain a lot of recipients that are bounced for various obvious reasons (mailbox does not exist any more is very common). Then thanks to the new AUTO-CLEAN feature, you get an exact report of which recipient you'll have to remove from your mailing list. Maintenance of such mailing list is then extremely easy. Furthemore, if you plan to use the new greylse daemon, then it will be able to work on the basis of this auto-clean black list to intercept any email that your mailing list engine can send to problematic recipients. Then here, it greatly help to not emit tons of bounces over internet everywhere, that drive you to have your sending servers/ip addresses blacklisted by DNSRBLs services. To conclude, AUTO-CLEAN ELSE feature + GreyLSE blacklisting feature is a very interesting association allowing you to automatically maintain your mailing lists (even if those features can be used for more general purposes)
- remain reporting Web UI window to implement, with CSV export
(Trunk r1978):
- greylse load other settings from database like greylisting delay, max_age sender email address is kept in autowhitelist, number of emails before autowhitelist
(Trunk r1977):
- created autoclean_policies, autoclean_reports, autoclean_bl database tables. They are needed for the AUTO-CLEAN reporting and blacklisting processing of the ELSE
- updated trig_smtp() SQL method to implement AUTO-CLEAN processing
(Trunk r1976):
- Bug fix: an issue prevented the content of the search result to be displayed in the Search Result window because of wrong if conditions that introduced bad HTML generated code
(Trunk r1975):
- added greylse_clients SQL table. It will maintain the list of postfix clients authorized to connect to the greylse instances
(Trunk r1974):
- parameters for the running greylse instance are now loaded from the database (port to listen on, socket timeout, maximum number of childs, defaul postfix policy in case of error like database not available, ...)
- as soon as the greylse daemon is launched, it query the database and creates its configuration by default in the greylses SQL table if the instance is not already existing in the database
- updated the INSTALL file consequently
(Trunk r1973):
- added new greylses SQL table
(Trunk r1972):
- Added new checkgreylse() SQL method. It returns the ID of a GreyLSE instance and create the requested instance with default parameters if it doesn't exist in the greylses table.
(Trunk r1971):
- This is the first publication on Sourceforge of the greylse daemon C/C++ code. It is considered as a first draft of something highly experimental.
- The greylse is an implementation of a Postfix Policy Server. It will use the E-LSE (E-mail Log Search Engine) tool, database, and web 2.0 interface
(Trunk r1970):
- added the new (but empty for now) greylse() SQL method. This method will contain the implementation of my Postfix Policy Server for Greylisting. For now, this method always returns "dunno" when called by the GreyLSE Policy Server. My postfix Policy Server will have all its core logic directly implemented in the database. The C/C++ GreyLSE daemon I'll also provide will just wait for postfix connections, send all what it receives on its socket to the database engine through a select call to the greylse() SQL method, and return the answer as it is provided by the resulting output of the greylse() SQL method on its socket. So to add features to my Policy Server, it'll just needed to update the greylse() SQL code inside the database.
(Trunk r1969):
- added EXCEPTION blocks to checkhost*() SQL methods to avoid potential race conditions
(Trunk r1968):
- added an EXCEPTION block to SQL function checkessession() to avoid possible race condition
(Trunk r1967):
- new alerts are displayed in the email details window in case it's suspected the email is backscattering or HELO/EHLO is not conform to sender email address domain name
(Trunk r1966):
- bug fix: possible race condition in SQL method checkdomaintree(), solved by adding an EXCEPTION block
(Trunk r1965):
- updated trig_iis_svc() SQL function to handle DOMAIN/user parsing when Kerberos is used
(Trunk r1964):
- adjusted some column widths having fixed sizes by replacing flex: lines by width: lines. This has been done for The extended search results and noqueue search results windows
(Trunk r1963):
- updated postgresql crontab including call to tasks_5min() SQL function
(Trunk r1962):
- created tasks_5min() SQL function to be called every 5 minutes by crontab. Some operations are now done there instead of being done in stats_5min()
- added a check in cookies parsing code in two exchange server parsing SQL functions. The DB could be blocked if cookie variable was empty.
(Trunk r1961):
- added new tables for Postgrey log parsing: postgrey_actions, postgrey_reasons, postgrey_wait
- added new SQL functions for Postgrey log parsing, updated postfix dispatcher SQL function to handle postgrey logs
(Trunk r1960):
- Added a Clear button to 'My E-mail Search Requests' to reset the grid
(Trunk r1959):
- updated noqueueseach.php script to deal with new search filters introduced in the web UI NOQUEUE List
(Trunk r1958):
- updated noqueuelist() SQL function to allow doing searches using operator 'domain' for 'from' and 'to' fields
(Trunk r1957):
- updated noqueuelist() SQL function to limit its output to 2000 rows, and deal with new filter fields introduced in the web UI (user group, server role, operator combobox for extended status)
(Trunk r1956):
- Updated NOQUEUE List Web UI to include some fields introduced recently in the Extended Search Web UI: filter by User group, Server role, changed Info Pattern to be Ext. Status, added the combolist Operator for Ext. Status, removed server field
(Trunk r1955):
- updated check and insert in table qmgr_uniqinfos to match defined primary key in trig_es_trck() SQL function
(Trunk r1954):
- bug fix: attachedserverobjectslist.php: added " to avoid PHP Parse error: syntax error, unexpected T_ENCAPSED_AND_WHITESPACE, expecting T_STRING or T_VARIABLE or T_NUM_STRING
(Trunk r1953):
- reset the last_qmgr_id values of the server table. that is needed because the stats_5min() has been updated to use qmgr_uniqinfos table
(Trunk r1952):
- qidslist() SQL function now makes use of qmgr_uniqinfos table instead of qmgr table
(Trunk r1951):
- queuelist() and sendersqueuelist() SQL functions now make use of qmgr_uniqinfos table instead of qmgr table (2x faster)
(Trunk r1950):
- updated stats_5min() SQL function to make it use the qmgr_uniqinfos table instead of the qmgr table
(Trunk r1949):
- updated the cleanner() SQL function to keep only the last 10 days of NDRs in table ndrswaitarea
(Trunk r1948):
- Bug fix: the content NDR when there are bounced emails was not displayed any more in the search result window, due to a bad variable name.
(Trunk r1947):
- bug fix: in dosearch.php: $subject could be wrongly encoded, so the results were not displayed in the search result window (no content in the window). This happened when subject contained ' symbol
(Trunk r1946):
- NDR2DB bash script: added some comments about configurable variables and values
(Trunk r1945):
- Added code to write operations done by NDR2DB and captureNDR bash scripts in log files
- updated for Postfix >=2.9.x long queue IDs
(Trunk r1944):
- Removed a mistyped caracter in the file, which made PostgreSQL generating an error when importing this SQL file in the database
(Trunk r1943):
- bug fix: in trig_es_trck() SQL function: data were recorded in qmgr table, but not in qmgr_uniqinfos table, which prevented all Exchange emails to appear in the extended search results
(Trunk r1942):
- added 2 additionnal reports to getstats() SQL function. Those reports give Top 25 Bounced/Deferred by sending systems. Those reports are very interesting to find what Ip addresses are the sender of MASS emails to recipients that are bounced or deferred. Those reports are available to the software administrators for now.
(Trunk r1941):
- updated getstats.php script to use the new maxreturn field of the reporttypes table. This field LIMIT the number of rows returned by the SQL queries providing reporting.
(Trunk r1940):
- added 4 additionnal reports to getstats() SQL function. Those reports give Top 25 Bounced/Deferred by RFC821/RFC822 senders. Those reports are very interesting to find who is the sender of MASS emails to recipients that are bounced or deferred. Those reports are available to the software administrators for now.
- the 4 "Top 25 Rejected ..." reports can now be used by users of the software not being administrators
(Trunk r1939):
- database schema patch file to go from version 0.9.15 to 0.9.16 of the tool
- added maxreturn column to reporttypes table. This column contains the maximum number of rows that should be returned for the reports (that number is used in the LIMIT clause of the SQL request)
- added 4 additionnal reports to the table reporttypes
0.9.15:
-------
Summary:
- The tool can now parse Postfix versions >=2.9.x logs, containing long queue IDs
- All calls to the database are now done using PDO, instead of direct PostgreSQL dedicated methods. So you have to ensure php-pdo is installed on your server
- the database is opened using each time a new connection using PDO. That means the web UI should not be blocked as before until a SQL request finish. You'll have to ensure the number of simultaneous PostgreSQL connections is set accordingly (file postgresql.conf)
- The "Info" tab show a new "Password update" section allowing any user to change his password. REMEMBER: You've to ensure you use the software ONLY via HTTPS
- in the "Statistics" tab, several new reports have been added (useable only by users defined as administrators of the tool for now) about rejected emails: that's the ¨TOP 25 reports in the list
- The Extended Search has been extended to allow you to define additional filters like the User Group the user using the tool is member of, and the Server role. Previously, searches were performed in all groups and for all server roles defined for the user.
- started to rework the "Log re-builder" tab to implement the new "Raw Logs" feature. This feature will allow the users to download an archive of the logs they will have selected on the available servers. This is a work in progress. For now, the Raw Logs Web UI is disabled in the tool. BUT, the necessary has been implemented in the database: raw logs are stored in a table. This table is round robin and data have a lifetime of 14 days. Waiting the GUI is coded, if you now SQ commands well, you can already get the raw logs by yourselves. Raw Logs are not like the Log rebuilder: Log rebuilder reconstruct the data Using a Postfix log format according data processed and distributed in the various tables of the database. Raw Logs give you logs as they are received by the database before any processing.
- the "Logs" sub-tab in the "Manage" tab has been reworked to include a paging tool bar
- the "Billing reports" graphs have been extended to implement an auto-refresh button. Also, when showing the data grid (button "Show grid"), the grid window can be resized AND download as CSV feature as been implemented. Then some data fields have been added to all of that like incoming rejected emails (inrejected), outgoing deferred emails (outdeferred), outgoing bounced emails (outbounced) for the domains you've selected in the report.
- this version also have the premises of the implementation of the RTAAM, meaning Real Time Alerting and Action Module. For now, the job has been done at the Database level. It consists in maintaining statistics at mailbox level, with a resolution of 1 minute, during 2 hours roundrobin. The software had already statistics but at domain level. Here we go a step further. Basically, RTAAM goal is the near-real-time detection and alerting of issues on the email flows. It will be primarily designed to tell you if you are under sudden and massive SPAM attack, and who are the targets AND the attackers. The concept is working already in its basics and if you know well SQL commands, you can get information about attackers and targets already. The Web UI parts have to be implemented for the next steps. Also, something will be provided for integration with Nagios monitoring. The software is also actually able to warn you if the email resulting of your search is PHISING or SPOOFING. This will also be part of the RTAAM to receive alerts in real time about this threats.
- emails headers and subjects should be decoded more correctly if they are in various charsets. That was not the case before. You should see the difference in your search results, in the extended search results window for instance (but if you do a search based on the subject, you must write it in the fields under its encoded format for now)
- several important and minor bug fixes in the Database schema, functions and PHP code as usual. An example of some important ones are:
* the creation of a new qmgr_uniqinfos DB table to avoid a discrepancy in the results provided by the statistic graphs, and by the extended search
* the possibility to get emails without subject nor HELO in the search results (was not the case before) to also avoid a discrepancy
Details:
(Trunk r1938):
- added versionning information for v0.9.15 in README file
(Trunk r1937):
- created new versions table. This table contain information about versions of DB Schema and the software itself
(Trunk r1936):
- altered type column customdata of es_msgtrack table to be TEXT instead of VARCHAR
(Trunk r1935):
- reuse htmlentities() specifying 'UTF-8' where needed in dofullsearch.php and dosearch.php
(trunk r1934):
- updated dofullsearch.php script to allow searches with user group and server role criteria
- updated usergoups.php and roleslist.php scripts that now provide the same data list as previously, with also the ANY data having id 0 (the wildcard '*' in fact) if the $star parameter is set to true
- creation of two new stores (AnyRolesListStore.js and AnyUserGroupsStore.js) that provide the same data as RolesListStore.js and UserGroupsStore.js except they set the star parameter. Those stores will then have the '*' option listed in their data
- updated app.js to include both new stores AnyRolesListStore.js and AnyUserGroupsStore.js
- Updated "Transport maps Decisionnal layer" as well as "Send an E-mail" Web UIs in tab "Test tools" to display the first available server of the list for their combobox
- Updated "Extended Search" Web UI of tab "Search" to include the user group and server role comboboxes in the filters section
- Updated Search stores to include fields related to usergroup and serverrole
- updated MyGridPanel.js and ExtendedSearch.js to correctly handle new usergroup and serverrole fields
(Trunk r1933):
- fullsearch() SQL function has been redesigned to allow searches restricted by server role and user group, additionnaly to all other existing criteria
(Trunk r1932):
- renamed "Server group" by "Server role" in "Statistics" tab
(Trunk r1931):
- started to implement the Raw Logs feature in the "Log re-builder" tab. The WEB UI of this tab has been reworked a little to contain the rebuilding feature as before, and the raw logs new feature. For now this last one is disabled as it is just an implementation job starting
(Trunk r1930):
- Added a paging tool bar to navigate in logs
(Trunk r1929):
- Added needed code for pagingtoolbar management in "Logs" sub-tab under "Manage" tab
(Trunk r1928):
- "Logs" sub-tab under "Manage" tab: removed buffered directives as it seems to be not really nicely supported by ExtJS 4.1.3
(Trunk r1927):
- Bug fix: Subject of email as well as header should be correctly (well, better at least...) decoded when they are encoded in various charsets. Fix done when displaying "graphical view", "text view", as well as extended search results window
(Trunk r1926):
- logrebuilder.php updated to make use of PDO for database calls (side effects might be possible, giving regressions. If any are found, we'll fix them)
(Trunk r1925):
- drawmailflow.php updated to make use of PDO for database calls (side effects might be possible, giving regressions. If any are found, we'll fix them)
(Trunk r1924):
- dosearch.php updated to make use of PDO for database calls (side effects might be possible, giving regressions. If any are found, we'll fix them)
(Trunk r1923):
- dofullsearch.php updated to make use of PDO for database calls (side effects might be possible, giving regressions. If any are found, we'll fix them)
(Trunk r1922):
- bug fix: in fullsearch() SQL function. the left join in the SQL request was not working due to the fact a filter on the date for the corresponding table (cleanup_head) was not part of the join condition, but of the remaining request conditions
(Trunk r1921):
- database is now opened by the code in file common.php. That allowed code savings in all other source files.
(Trunk r1920):
- most PHP scripts updated to make use of PDO for database calls (side effects might be possible, giving regressions. If any are found, we'll fix them)
(Trunk r1919):
- For Billing Reports: added rejectedin, deferredout and bouncedout in graph report and in data grid; added download button do data grid; made data grid resizable, and wider when first displayed
(Trunk r1918):
- added inrejected, outdeferred and outbounced fields in ChartDataList.js model
(Trunk r1917):
- loadreportdata.php updated to load inrejected, outdeferred and outbounced data
(Trunk r1916):
- updated two generateReport() SQL functions to add bounced and deferred columns in the results
(Trunk r1915):
- provided updatepwd.php script that is used to update a user's password (by the user himself). REMEMBER: please never use the software if not over HTTPS
(Trunk r1914):
- correctly reset password update formular once password successfully changed
(Trunk r1913):
- Added an alert box once user changed his password successfully
(Trunk r1912):
- Added GUI elements and logic to allow user to change his own password
(Trunk r1911):
- updated fullsearch() SQL method to use qmgr_uniqinfos table instead of qmgr table
- bug fix: using left join with cleanup_head table to avoid not showing results when emails had no subject or no helo
(Trunk r1910):
- updated scripts to use PDO instead of direct postgreSQL primitives.
- updated SQL request providing results to handle OFFSET and LIMIT received as parameters in logs.php
(Trunk r1909):
- added PHP function setLastUserAccess(). This one is making use of PDO
(Trunk r1908):
- bug fix: implemented another method to record HELO and SUBJECT of an email in trig_cleanup() SQL function, because the latest update of this function caused a lot of duplicates in the table cleanup_head
(Trunk r1907):
- created table sendertraffic_monitor. This table will record per sender e-mail address statistics made according emails sent to recipients. It's basically the same as table usertraffic_monitor.
- updated usermonitor_cleanner() SQL function to clean table sendertraffic_monitor also
- updated bill_r_stats() SQL function to maintain per sender e-mail address statistics in table sendertraffic_monitor. By default, this table will record data with a resolution of 1 minute for each e-mail address, and for a lifetime of 2 hours roundrobin. Number of data in this table will be absolutely enormous if you're an ISP. This is an experimental feature to start working on real time alerting module of the tool (RTAAM: Real Time Alerting and Action Module). The goal is to send alerts when attacks on messaging systems are detected in real time, providing what e-mail address is the sender easyly. For example, this table should allow to rise an alert if number of bounced emails created by a sender is higher than a configured value (one bahavior of MASSIVE SPAM is that a lot of emails are sent to random e-mail address recipients, thus generating a lot of bounced or deferred emails in a very short time period). This table will also allow graphing of statistics like there is for now in the tool on a domain basis, but on an e-mail address basis.
- the table usertraffic_monitor is designed to allow detection of e-mail addresses under attack (receiving tons of emails sent by a spammer for example)
- the table sendertraffic_monitor is designed to allow detection of sender e-mail addresses being the attacker (the spammer itself for example)
(Trunk r1906):
- created table usertraffic_monitor. This table will record per e-mail address statistics. It's basically the same as table billing_stats
- bug fix: function timestamp_modulo introduced a time zone shift on systems having a non GMT UTC time
- updated bill_s_stats() and bill_r_stats() SQL functions to maintain per e-mail address statistics in table usertraffic_monitor. By default, this table will record data with a resolution of 1 minute for each e-mail address, and for a lifetime of 2 hours roundrobin. Number of data in this table will be absolutely enormous if you're an ISP. This is an experimental feature to start working on real time alerting module of the tool. The goal is to send alerts when attacks on messaging systems are detected in real time, providing what e-mail address is the sender easyly. For example, this table should allow to rise an alert if number of bounced emails created by a sender is higher than a configured value (one bahavior of MASSIVE SPAM is that a lot of emails are sent to random e-mail address recipients, thus generating a lot of bounced or deferred emails in a very short time period). This table will also allow graphing of statistics like there is for now in the tool on a domain basis, but on an e-mail address basis
(Trunk r1905):
- added call to usermonitor_cleanner() SQL function in crontab
- updated frequency of calls to function cleanner(). this must be called each 15 minutes instead of once a day for ISPs
(Trunk r1904):
- added syslogtag column to rawlogs table
- updated trigger_parser* SQL functions to record NEW.syslogtag in the rawlogs table
(Trunk r1903):
- all postfix parsing functions have been updated to support Postfix Long Queue IDS. Postfix <=2.8.x and >=2.9.x logs can now be parsed
(Trunk r1902):
- extended qid and bounceqid columns of table ndrswaitarea from char(11) to char(32) to make it ready processing long queue ids from postfix >=2.9.x
(Trunk r1901):
- bug fix: updated config table: records logheader and logsubject could be set to 'true' or 'false' instead of '1' or '0', which prevented the parsing by the database to work. Also updated very old and now unused cleanup() function that inserted those wrong values in the table
- bug fix: updated trig_smtpd() and trig_smtp() SQL functions to allow parsing of postfix logs containing IPV6 addresses. The parsing of the logs by the database could hang before this
(Trunk r1900):
- Bug fix: in SQL function getstats(). Now using new qmgr_uniqinfos table instead of the qmgr table. Reports show now... different results (and fixed results)
(Trunk r1899):
- Added a bigserial column at creation of table qmgr_uniqinfos
(Trunk r1898):
- created table qmgr_uniqinfos. This table is a kind of sister of the qmgr table. The difference is that it contains only a unique qmgr record for any email transaction, whereas the qmgr table contains all records of any email transactions (there are several records in the case Postfix try to re-send emails to recipients after a defer). This table is primarily there to fix statistics graphs which can display wrong data because counting several times the same fields for emails having defer and retries. It will also be a litlle bit faster due to the fact it will contain less records as the qmgr table
- function trig_qmgr() has been updated consequently to the addition of the new qmgr_uniqinfos table
- added IF EXISTS to DROP FUNCTION commands in XItool_MailManager_SQLproc
(Trunk r1897):
- bug fix: in function trig_cleanup(). Implemented a better regex to find the helo field
(Trunk r1896):
- Bug fix: in trig_cleanup() SQL function: if subject of email not existing, the HELO was not registered in the DB
(Trunk r1895):
- updated type of columns totalrequesttime, totaldcrequestlatency, totaldcrequestcount to bigint instead of int4 for table es_ews
(Trunk r1894):
- Updated SQL function trig_es_ews(). Replaced some int4 by bigint in splitting operation
(Trunk r1893):
- type of column throttlingrequesttype for table es_ews updated to varchar(128);
- type of column msginfo for table es_msgtrack updated to text;
(Trunk r1892):
- Billing report graphs now have the auto-refresh feature (every 20 seconds)
(Trunk r1891):
- implemented auto-refresh functions in ChartDataListStore
(Trunk r1890):
- updated index.html for version 0.9.15
(Trunk r1889):
- Defined x and y variable to move the "My Search Requests" window at a better position on screen on application start
(Trunk r1888):
- updated MyContainer.js to automatically open the "My Search Requests" window onBeforeRender. This window is also placed "ideally" on the screen
(Trunk r1887):
- Simplified Viewport: It opens only the main window of the tool, and not the "My Search Requests" window any more, opened once the main window is loaded by itself
(Trunk r1886):
- file regrouping all GUIs used to search in tab "Search" (splitting of code)
(Trunk r1885):
- provided javascript header files for new "Raw Logs" coming feature
(Trunk r1884):
- provided RawLogsStore store that will be used for the new RAW LOGS export feature (RwaLogsStore.js)
(Trunk r1883):
- provided RawLogs model used by RawLogsStore (RawLogs.js)
(Trunk r1882):
- created table server_role_rawlog_types doing the link between server roles and raw log types
(Trunk r1881):
- removed column log_id from rawlogs table which is useless
(Trunk r1880):
- created new rawlog_types and rawlogs table (for the implementation of upcoming "raw logs" feature)
- updated several SQL functions to record any log line in the rawlogs table. Incoming logs are registered as they come in this table, before any processing.
- updated the cleanner() SQL function to clean rawlogs table (default live time is 14 days)
(Trunk r1879):
- revised all SQL functions to make some of them STABLE (volatility category)
(Trunk r1878):
- extended column reference of es_msgtrack table to varchar(1024)
- created 4 new reports in table reporttypes, about rejected emails (NOQUEUE from postfix servers)
(Trunk r1877):
- Bug fix: the createdby ID was not set properly when a user created a new report in tab "Manage", sub-tab "Billing reports". The consequence was that the report wasn't displayed in the list of the available ones in the GUI
0.9.14:
-------
(Trunk r1876):
- added versionning information for v0.9.14 in README file
(Trunk r1875):
- updated version number to 0.9.14 in adequate php/html files
(Trunk r1874):
- added some cast(as bigint) in SQL requests performed in getstats function
(Trunk r1873):
- bug fix: in getstats function, first case '12', removed an unused table that made the SQL request extremely slow
(Trunk r1872):
- queuelist function updated to also return the rfc822from field
(Trunk r1871):
- updated to show RFC822 FROM as an additional column in the email queue window
(Trunk r1870):
- bug fix: users can only see the billing reports created by the users in the same usergroups they are member of.
(Trunk r1869):
- servers available in the billing reports definition are now matching with servers a user have the right to see according his permissions
(Trunk r1868):
- Updated two report names in reporttypes table (for bounced and deferred)
(Trunk r1867):
- all SQL commands used to generate reports have been moved and reworked in the getstats SQL function
(Trunk r1866):
- bug fix: both queuelist and sendersqueuelist functions have been fixed to provide a correct view of queued emails on servers and about domains users of the tool have permissions
(Trunk r1865):
- created getstats function. It's now allowing the tool to generate statistics according what are each user permissions (what user account can access what server and what domains on them)
(Trunk r1864):
- Extended column msginfo of table es_msgtrack to varchar(2048)
(Trunk r1863):
- updated Exchange server parsing functions to deal with IPV6 addresses
(Trunk r1862):
- added a 'comment' column to the servers table
- extended the 'value' column of the iis_cookie_pairs table to 768 chars
- updates in stored functions
(Trunk r1861):
- added the possibility to update the IP addresses of SMTP servers
- Added the "Domain" option in the matching store (that will allow the users to perform extended searches using a domain name instead of an e-mail address for instance)
(Trunk r1860):
- Bug fix: removed handling of #015 code in exchange server log processing functions. That was dur to a wrong configuration of the windows rsyslog agent.
(Trunk r1859):
- added parsing of postfix/pipe log lines
(Trunk r1858):
- Drawing of the XItools E-mail Log Search Engine (E-LSE) Database Schema, for version 0.9.14 of the tool, DB rev 955 PostgreSQL 9.2.3
(Trunk r1857):
- Added an index to the domain_id column of the emails table
- updated fullsearch function to allow searching based on a domain (domain_id) criteria
- updated trig_es_rca and trig_es_ews functions to check the resulting number of fields in parsed lines. It should match a certain numebr of columns and before, the simple fact this number of columns could differ stopped the processing of some kind of exchange server logs.
(Trunk r1856):
- for huge Microsoft Exchange Server log parsing, the database was slow on a specific SQL request executed every 5 minutes by crontab. The updates provided there contains some optimizations to avoid that. A column has been added to a table, some indexes have been created, some SQL functions have been updated consequently, and an index has been removed because it is far less efficient than the updates provided there. In a huge customer production environnement, the execution of the SQL function now takes 0.5 second when it used 257 seconds previously!
(Trunk r1855):
- Completed versionning information for v0.9.13
0.9.13:
(Trunk r1854):
- Fixes in generateReport SQL functions
(Trunk r1853):
- added versionning information for v0.9.13 in README file
(Trunk r1852):
- This is version 0.9.13 of the tool
- In the queue list window, the drop-down list displays the list of sender e-mail addresses of queued emails. But the "delete" feature is still not implemented for now.
- "Resolution tools" tab has been renamed "test tools"
- in the "test tools" tab, the BIG IP F5 tool is now implemented. A user can see only the BIG IP F5 assigned to his groups
- a new test tool has been added to the "Test Tools" tab: you can now use it to make any postfix server send an email to a destination. It generate a telnet session on the target server using SNMP. That can be used to test SMTP connectivity between servers
- When generating a billing report graph, it's now possible to select a resolution of 15 minutes.
- A new sub-tab for managing BIG IP F5 loadbalancers in the tool has been added under the "Manage" tab
(Trunk r1851):
- added sendersqueuelist function
(Trunk r1850):
- Creation of the BIG IP F5 tables
(Trunk r1849):
- addf5.php add a BIG IP F5 loadbalancer to a group
(Trunk r1848):
- the loading icon have a nice fade out and disappear when the application is ready to be displayed
(Trunk r1847):
- attachedf5list.php gives the list of BIG IP F5 attached to a group
(Trunk r1846):
- deleteattachedf5.php add or remove a BIG IP F5 object from a group
(Trunk r1845):
- deletef5.php remove the data of a BIG IP F5 from the database
(Trunk r1844):
- deleteserver.php delete a mail server from the database. All data related to this server are then erased from the DB, and this operation can take a lot of time depending of the amount of data
(Trunk r1843):
- emailtest.php generates a telnet session on port 25 on a remote postfix server using the snmp_emailtest
(Trunk r1842):
- f5resolver.php script contact a BIG IP F5 loadbalancer via SSH and run commands on it then display the result. This script is used to know which client IP address (the user) is loadbalanced on which node behind the loadbalancer
- f5list.php provide a list of registered BIG IP F5 loadbalancers
(Trunk r1841):
- sendersqueuelist.php script used to provide the list of sender e-mail addresses of e-mails queued on a postfix server
(Trunk r1840):
- updatef5.php script used to update the settings of a big ip f5 loadbalancer object in the database
(Trunk r1839):
- Added the BIG IP F5 loadbalancer icon
- Added snmp_emailtest script that use SNMP to contact any postfix server for doing a telnet session test on port 25 when user request it via the tool
(Trunk r1838):
- added versionning information for v0.9.12
0.9.12:
(Trunk r1837):
- This is version 0.9.12 of the tool
- a lot of very long SQL queries that were in the PHP files have been deported as stored SQL functions in the DB itself. It's faster, and will save numerous updates of the PHP files in the futur.
- in the "info" tab of the web UI, when double-clicking on a postfix server, the queue list window now show a drop down list, which is empty for now. This drop down list is to prepare a new and futur feature of the tool that will allow you to delete all queued e-mails related to a specific sender e-mail addres. The queue list window also display a little bit more details for each queued e-mail like the number of tries the server did to send the e-mail to each recipient. The number of transactions that can be displayed in the queue list window is now set to 1000 maximum.
- in the "manage" tab, and in the "billing reports" sub tab, when generating a report ("report" button clicked), the resulting graph could be longer than the default timeout to be displayed and in such case no graph was dipslayed. This timeout has been now set to 300000 like for the other stores.
- some little updates also for exchange server logs parsing, in the SQL stored functions
- in the "resolution tools" tab, there is one that is not functionnal: the Big IP F5 Resolver. The script making it working will be provided for version 0.9.13 of the tool. For now, the result window that will appear when this resolution tool will be used has been updated. The goal of the Big IP F5 Resolver is to query your Big IP F5 load balancers to know where it sent the user requests (on which node behind the load balancer) according the user's IP address. All of this is coming for version 0.9.13 of the tool.
(Trunk r1836):
- queue list function output is now limited to 1000 lines
- added additional checks in exchange server logs parsing functions
- other minor updates
(Trunk r1835):
- creation of the smtp_cache table
(Trunk r1834):
- Added several SQL functions to handle several big SQL requests instead of building them in the PHP source code files
(Trunk r1833):
- database schema patch file to go from version 0.9.11 to 0.9.12 of the tool
- added missed server roles in server_role table
(Trunk r1832):
- typo fixes and small precisions added in the README file
(Trunk r1831):
- updated README file to include versionning information for version 0.9.11 of the tool
0.9.11:
(Trunk r1830):
- updated INSTALL file to include information about Microsoft Exchange Server log parsing
(Trunk r1829):
- EXCHANGE_SRV_LOGS: added description of all exchange server log files the tool is able to parse (needed log structure and format)
(Trunk r1828):
- This is the README file (EXCHANGE_SRV_LOGS) explaining VERY important things for Exchange Server Logs parsing
(Trunk r1827):
- This is version 0.9.11 of the tool
- This version includes Exchange Server Logs parsing capabilities for the first time but be careful: you have to set the colums in your exchange server log files to be EXACTLY in the order needed for the tool. You must also include in your log files the columns requested by the tool. So, remember TWO THINGS: include all necessary fields, and set them in the right order!!! You can see what are the required fields and their order by putting your hands in the code of the SQL stored procedures file (XItool_MailManager_SQLproc), and by trying to find all the functions making the Exchange Server Logs parsing. A new EXCHANGE_SRV_LOGS readme-like file will be provided soon to explain this clearly.
- due to the fact this tool can also be used by the customers of a messaging services provider, and because several customers (users) can be hosted on the same e-mail servers, you know have the possibility to restrict usage of the tool by users/servers/DOMAINS relations, the containers being the groups).
- the info tab WEB UI has been updated: if you double-click on a POSTFIX server in the servers list, the current server QUEUE will be displayed in a new window; Then by double-clicking on a queued e-mail, you will get all the details regarding it (which is equivalent to the search by QID feature). This feature is entirely relying on what is in the database, your postfix servers are not contacted to get the list of queued e-mails.
- various UI list components now allow you to do MULTIPLE SELECT instead of selecting just one element. That makes Drag&Drop easier between two lists for instance
(Trunk r1826):
- added all functions for version 0.9.11 doing Exchange Server Logs parsing
- some updates on postfix parsing functions
- implemented management of pfx_cache table
(Trunk r1825):
- typo fixes in INSTALL file
- ExtJS useable version fixes in INSTALL file
(Trunk r1824):
- added all changes until version 0.9.10 in the README file
(Trunk r1823):
- added this empty README file
(Trunk r1822):
- updated header of the INSTALL file
(Trunk r1821):
- Fix: CREATE OR REPLACE FUNCTION instead of CREATE FUNCTION
(Trunk r1820):
- database schema patch file to go from version 0.9.7 to 0.9.11 of the tool
0.9.10:
(Trunk r1819):
- This is version 0.9.10 of the tool
- updated extended search and log rebuilder to specify subject using operators
- moved some methods from controller/Search.js directly at the level of the UI code files
- continued reorganization of Search Result Window to display a lot of QIDs
0.9.9:
(Trunk r1818):
- This is version 0.9.9 of the tool
- reworked the E-mail Search result window. Reworked corresponding code. This rework allow for the display of a lot more QIDs inside of a same e-mail, this is also faster.
0.9.8:
(Trunk r1817):
- This is version 0.9.8 of the tool
- Added an animated picture when loading the tool (because again, this is a big web application that can need a lot of time to start, and during the loading, the browser stayed white...)
- added a CSV/Excel export capability. Added a "download" button in several grids
0.9.7:
(Trunk r1816):
- Added the exporter source code found on Sencha and contributed by somebody else than me
- it's to be clear all this source code for the exporter is not created, made, nor maintained by me
(Trunk r1815):
- Fix: replaced and by && in a if test
(Trunk r1814):
- Added a section on requirements for postfix servers in INSTALL file
- provided snmp_exec script that will be used by the tool
(Trunk r1813):
- implemented function trig_postsuper
- little other updates
(Trunk r1812):
- database schema patch file to go from version 0.9.6 to 0.9.7 of the tool
(Trunk r1811):
- This is version 0.9.7 of the tool
- Added a new billing flag to the users. When this flag is set, he can access to the volume graphs. before, everybody could access them. The GUI for user addition/edition has been updated consequently.
- GUI for searching by QID or Message ID has been splitted into panels under the "Search" tab
- reworked the way results are sent to the browser/GUI to enhance perfs a little bit, for extended search and noqueue search
- added code for resolution tools. DNS resolver now uses php_ssh2 library (this library is owned by his own author)
- added a BIG IP F5 resolver. GUI is there. Script to provide results is not at this time, but that will come later
- Added a new feature: ACLs management on a new "ACLs" tab. This big new feature is still under development but it will allow you to manage Postfix policies 'like in CheckPoint FW NG', that will be pushed on impacted servers. For now, the policy management concept implemented allow you to manage the content of the mynetworks Postfix configuration directive. In the futur, other kind of directives might be also managed. Even if ACls (or policies) are not completely operationnal at this time, you can already create your objects. For this to work, a set of new DB tables have been introduced. Donùt forget to run the script to update the DB for this version 0.9.7.
0.9.6:
(Trunk r1810):
- updated the SQL template for version 0.9.6 of the tool
(Trunk r1809):
- This is version 0.9.6 of the tool
- enhanced GUI
- added feature: a new satistics tab in the GUI with more than 20 available reports ("top 20 reports")
- added features: in the "manage" tab. It's now possible to manage the tool, what user can access and search what servers. There's also another reporting tool in the manage tab, allowing you to create "volume" graphs or your e-mail traffic, for billing based on that for example. Also, it's now possible to see the logs generated by the tool itself for tracability of all the actions.
- bug fix: when searching using the extended search, based on the e-mail address recipients, the search was done using the "RCPT_TO" and not the "TO". Now, the search is done on both with a OR condition.
(Trunk r1808):
- Fixed a regression in the stats functions, to properly handle e-mails having status deliverable or undeliverable
(Trunk r1807):
- Fixed a regression: lmtp processing was removed in function trig_smtp during previous commit. It is back again
(Trunk r1806):
- added the sequence part_id
(Trunk r1805):
- database schema patch file to go from version 0.9.5 to 0.9.6 of the tool
- updated stored procedures
0.9.5:
(Trunk r1804):
- database schema patch file to go from version 0.9.0 to 0.9.5 of the tool
(Trunk r1803):
- This is version 0.9.5 of the tool
- Redesigned the Web User Interface. All tabs are now in a dedicated window
- cleanning of scripts (removed old unsued functions that were commented out)
- updated the servers grids to show server IPs. Same in "Servers" tab under "Manage"
- added scripts to display logs, manage users and servers attached to groups (but logic still needs to be added in the WUI for that)
0.9.0:
(Trunk r1802):
- Each script now make checks against the database according the rights and permissions of the current user
- Start to code the "Manage" tab. There are 3 sub-tabs now in this tab: "Groups" to manage groups, "Users" to manage users and "Servers" to manage Postfix servers. In this commit of the tool, it is possible to update the roles of the postfix servers, and it is possible to manage the users (Add, Edit, delete). The WUI for "Groups" is there but there is almost nothing behind for now: it is just possible to create, update and edit a group. Users and Servers cannot be attached to a group at this time.
- Added a Monitoring directory in the Tools directory: that will contains Cacti and Nagios configuration/template files in the futur.
(Trunk r1801):
- finished POSTFIX NDRS CAPTURE section in INSTALL file
(Trunk r1800):
- added transport file: contains configuration directives that must be added to the /etc/postfix/transport file
- added main.cf file: contains configuration directives that must be added to the /etc/postfix/main.cf file
- updated header_checks file
(Trunk r1799):
- This is version 0.9.0 of the X-Itools Mail Manager tool
- Added "Information" tab showing some user information (login ID, date of creation and last connection, ...) as well as the list of Postfix servers the user will be able to query
- Now when a search is done and email transactions are found and displayed in the "Search Result" window, this window will possibly display a new section in the "Text view" and also in the "Graphical view" showing various alerte in case inconsistencies are found in the e-mail header, like Phishing or Spoofing alerts
- Added the "Manage" tab that will be useable by the Tool administrators to manage all aspects of the tool with the GUI instead of typing SQL requests directly in the database like today. This tab is disabled for now because there is nothing in it. This feature implementation is in progress.
- The different tab panels have been reduced in size to be nicer. This is the begining of a review of the way the GUI is designed.
(Trunk r1798):
- database schema patch file to go from version 0.8.5 to 0.9.0 of the tool
0.8.5:
(Trunk r1797):
- added two default email addresses in table emails
(Trunk r1796):
- This is version 0.8.5 of the X-Itools Mail Manager tool
- Simple search tab: added the possibility to do simple search using a QID
- Extended search tab: added the possibility to search with more criterias, also added operators for some criterias (search by Exact match, Contains, Does not contains, Different)
- Added new tab "NOQUEUE list" to add the possibility to search in aborted e-mail transactions
- Added new tab "Log re-Builder" to add the possibility to rebuild a log file of postfix logs, based on the information stored in the database
- Added new tab "Resolution tool" to add futur features like DNS resolution tool, Postfix transport map checker, ... The features of this tab are not working for now in this release
- When an e-mail transaction is displayed in the "Search Result" window, added the new "Header" section to display the e-mail header captured by the tool. E-mail headers are captured if the logheader variable is set to true in the database table "config". By default this new "Header" section is closed. Open it by clicking on it.
(Trunk r1795):
- added a drop of the ndrs table in XItool_MailManager_0.7.4_0.8.5.out
- updated XItool_MailManager_SQLproc for version 0.8.5
(Trunk r1794):
- database schema patch file to go from version 0.7.4 to 0.8.5 of the tool
0.7.4:
(Trunk r1793):
- updated the UPDATE THE TOOL section, for database updates
(Trunk r1792):
- added pgpass file as .pgpass configuration example
- updated INSTALL file as a consequence
(Trunk r1791):
- removed some empty lines to avoid "image corrupted" when going to Graphical View of E-mail transactions
(Trunk r1790):
- Forgot to commit the drawmailflow.php file
(Trunk r1789):
- version 0.7.4 of the X-Itools Mail Manager tool
- added message-id import feature
- the "My Seach Requests" became a separated ExtJS window
- added the automatic e-mail flow network diagram generation feature
(Trunk r1788):
- Example of rsyslog.conf file to be installed on any Postfix servers you want the logs from
(Trunk r1787):
- Example of rsyslog.conf file to be installed on any Postfix servers you want the logs from
(Trunk r1786):
- updated stored procedures to allow processing of postfix/lmtp log lines
(Trunk r1785):
- database schema patch file to go from version 0.6.4 to 0.7.4 of the tool
0.6.4:
(Trunk r1784):
- Thanks to this new set of stored procedures, postfix log parsing can now be done "in real time" directly by the PostgreSQL database.
- Basically, Rsyslog must now be used to send the logs directly to the database.
(Trunk r1783):
- added crontab to be installed for postgres user on the database server
- added crontab to be installed for root user on any postfix server where NDR capture is required
- added the captureNDR script to be installed in /usr/local/bin on any postfix server where NDR capture is required
- added the NDR2DB script to be installed in /usr/local/bin on any postfix server where NDR capture is required
- added the aliases file containing the lines that must be copy/pasted to the end of the /etc/aliases file on any postfix server
(Trunk r1782):
- typo fixes in INSTALL file
(Trunk r1781):
- typo fixes in INSTALL file
(Trunk r1780):
- typo fixes and apache web server default directory path fixes in INSTALL file
(Trunk r1779):
- This is a first try to code the X-Itools Mail Manager web application in ExtJS 4.1.x
- The first version of the web client made entirely in HTML in 2004 was never publicly published.
- The code published here under several aspects is not functionnal but this is a good base to go to a powerfull ExtJS application for e-mail tracking
< 0.6.4:
(Trunk r1778):
- moved some database files in PG92 directory, as they are made for PostgreSQL 9.2.x
(Trunk r1777):
- This is the PostgreSQL database schema of the X-Itools Mail Manager tool. This schema has been ported to PostgreSQL 9.2.x
(Trunk r1776):
- This is a C version of the Mail manager, done in February 2004. The bash version was running fine but it was extremely slow. This C version must of course be compiled, and it is much faster.
(Trunk r1775):
- deported all bash script processing for log parsing as SQL stored procedures to enhance perfs.
(Trunk r1774):
- PostgreSQL stored procedures needed for the postfix log parser bash script
- Contains the checkqid procedure.
(Trunk r1773):
- This version of the Mail Manager bash script is now using only stored procedures in PostgreSQL for all the parsing. Remains there only the instructions to read each line of the log file and call the stored procedure with the read line as parameter.
(Trunk r1772):
- one of the problem of this script was that it was extremely slow to parse postfix log files (the /var/log/maillog file). I tried to implement all the parsing process in the database as pl/pgSQL stored procedures.
(Trunk r1771):
- First version of a postfix maillog parser I wrote in january 2004 for my own needs. It was kept in my private repository. Now, I publish it as the project reach recently a new dimension.
- This version has been written in bash script and make use of a PostgreSQL database
Download Latest Version
x-itools_ELSE_0.9.19.tar.gz (37.0 MB)
